Collecting data with Twitter API - Welcome to new things

I tried to get data from Twitter using the Twitter API to see if I could use it to learn about trends in the world.

As it turns out, I had heard rumors that the Twitter API was too restrictive, but it was really too tight to be useful...

Since I have done as far as acquiring data via Twitter API, I would like to note it as a memo on how to use Twitter API.

Introduction.

The procedure is based on the following article.

qiita.com

The following is a rough summary of the procedure and points to note.

A Twitter account is required to use the Twitter API
Twitter account must be registered with a phone number.
Register as a Developer with your Twitter account.
Register your app on the Developer site
If you just want to use the Twitter API, turn off the "Enable Sign in with Twitter" checkbox in the application registration.
Generate "API key", "API secret key", "Access token", and "Access token secret" in the application.

You can use the Twitter API with the following four pieces of information: API key, API secret key, Access token, and Access token secret.

Twitter API to collect data

App token screen on Developer site

API key", "API secret key", "Access token", and "Access token secret" are listed. When you want to create an application that calls the Twitter API using the token of a person logged in with a Twitter account, such as when you want to create a website that authenticates with Twitter, Check ON the "Enable Sign in with Twitter" checkbox in the application registration.

It's the same one that registers your application with Google when you create a Google-authenticated website.

treatment

The authentication process is complicated, but fortunately there is a JavaScript library called "twitter" that uses the Twitter API, so I used it.

To find out what APIs are available and how to use them, look at the Twitter API Reference.

https://developer.twitter.com/en/docs/api-reference-index

For example, you can retrieve Genji Yonezu's recent tweets at

import * as Twitter from 'twitter';

const tw = new Twitter({
    consumer_key: '<API key>',
    consumer_secret: '<API secret key>',
    access_token_key: '<Access token>',
    access_token_secret: '<Access token secret>'
});

function twitterAPIGet(client: any, url: string, params: any) {
    return new Promise((resolve, reject) => {
        client.get(url, params, (error: any, response: any) => {
            if (!error) {
                resolve(response);
            } else {
                reject(error);
            }
        });
    });
}

(async () => {
    try {

        const url = 'statuses/user_timeline';
        const params = {
            screen_name: 'hachi_08'
        };

        const res = await twitterAPIGet(tw, url, params);
        console.log(res);

    } catch (err) {
        console.log(err);
    }
})();

restriction

I had heard rumors, but the restrictions were tougher than I had imagined.

https://developer.twitter.com/en/docs/basics/rate-limits

The limit varies from API to API, but the tighter ones allow only 15 calls in 15 minutes, or one call per minute.

For example, if you try to get data on Genji Yonezu's followers (about 2.3 million) using the Twitter API, you can only get one call per minute, and each call can only get up to 200 followers.

2.3 million people ÷ 200 people ÷ 60 minutes ÷ 24 time = about 8 days

It will take....

Also, it is not as if you can acquire all the information if you only take the time to do so.

For example, if you try to retrieve all tweets for a given Twitter account, the API can only retrieve the most recent tweets.

From a cursory look at the API, it seems that Twitter does not intend to allow outsiders to collect information from Twitter, and that the Twitter API is mainly intended to be used to operate Twitter accounts that they have or have been authenticated to use.

Paid Plans

There are some features that you can use if you pay some money.

search

An API to search tweets is provided separately for a fee.

The API can only be used for rough searches by keywords, and is intended to be used like a newspaper article search.

https://developer.twitter.com/en/docs/tweets/search/overview

Things about me

Even when retrieving information about yourself, such as your tweets or your followers, the same API restrictions apply.

For users who are willing to pay to retrieve their own information, an API to retrieve their own information is provided separately for a fee.

However, while all of your own engagements can be retrieved, others are webhooked when the activity occurs, so it is not possible to retrieve information retroactively.

Instead of providing a means of acquisition, the data will be sent to you in real time, and you can save the rest yourself.

Impressions, etc.

I wanted to gather information from Twitter in various ways, but it is not realistic to gather information over a wide area using the Twitter API. Disappointing...