The inspirers behind the development of the statistics service for Tinkoff Investments were:

• article on Habré “What Tinkoff Investments are not saying” 
• analysis of the wishes of platform users
• an article about the calculation of commissions . 

What will be discussed?

• Only the applied part about development.
• Real knowledge and experience, which are very important in working with financial instruments.
• Overview of issues to work on

So, I want to calculate trade statistics and do it in a convenient way. 

Developing a statistics service step by step: 

1. Connection to Tinkoff Invest API
2. Drawing data from Tinkoff Invest API in a browser
3. Receiving brokerage reports and transactions
4. Calculation and output of information of interest
5. Conclusions and plans for the future

Connection to Tinkoff Invest API

To connect to the API, you can take any sdk from the documentation https://github.com/Tinkoff/investAPI#sdk . Or npm package ` tinkoff-sdk-grpc-js `. It is important that the package is updated to the latest version by the developers. Install

npm i tinkoff-sdk-grpc-js

Checking

const { createSdk } = require(‘tinkoff-sdk-grpc-js’);   // Token that can be obtained like this  const TOKEN = ‘YOURAPI’;   // The name of the application by which you can be found in the TCS logs. const appName = ‘tcsstat’;   const sdk = createSdk(TOKEN, appName); (async () => {     console.log(await sdk.users.getAccounts()); })();

Result: a list of your accounts will be displayed in the console. For example, let’s analyze the nuances:

• In the list of accounts there is an “Investment bank”, with which you cannot work using the API
• Please note that the fields come in camelCase, while in the documentation these fields are presented in under_score. 
• It will be like this everywhere, so you can’t just take and copy a field from the documentation.

Useful:

• You can find this code in the project branch

https://github.com/pskucherov/tcsstat/tree/step1 https://github.com/pskucherov/tcsstat/compare/step1   

Drawing data from Tinkoff Invest API in a browser

I took next.js and socket.io. This is not a strong recommendation, choose at your discretion. 

npx create-next-app@latest npm i socket.io socket.io-client

We immediately proceed to the friendship step next+socket+investapi, and see the Useful section of this step for all the details.  I’ll describe the details: 

• On the nodejs (server) side, there is a pages/api/investapi.js file. This is where we create the socket.io server and connect to investapi.
• On the browser (client) side, we connect to the server via a socket and request account data from the broker. 
• We receive data from the broker on the server, then send it to the client. When they are received on the client, they are displayed in the browser. 

Result:  in the browser console we can see information about accounts. That is, in the last step, we saw information about accounts in the server console (nodejs), in the current step, we transferred this information to the client (browser).

Now let’s make it so that you can select an account from the browser, and if there is no token, then an error is sent to the console. The work is simple and nothing new, so I give only links to commits

1. https://github.com/pskucherov/tcsstat/commit/7e1ac57061e5e971588479015b06d8814d6609a9
2. https://github.com/pskucherov/tcsstat/commit/b28ac973a57494f5232589b4cb6b9fb13b8af759 

Useful:

• How to make friends next and socket is described in detail here . 
• Friendship code next+socket+investapi:

https://github.com/pskucherov/tcsstat/commit/a443a4ac1bb4f0aa898f638128755fe7391ee381 For whom the above is difficult, then we remain at this stage and deal with the code. If you have questions – ask. https://github.com/pskucherov/tcsstat/tree/step2 https://github.com/pskucherov/tcsstat/compare/step1…step2

Receiving brokerage reports and transactions

There are three methods to receive brokerage reports and transactions

1. GetBrokerReport
2. GetDividendsForeignIssuer
3. GetOperationsByCursor

From the very beginning it is important to know: 

• The brokerage report is generated in the T-3 mode, i.e. trades are displayed there after their actual execution. 
• Accordingly, if you request this report for the last two days, it will be ready in three days. 
• To get transactions for the last days, we use the method for receiving transactions, but remember that their id and content may change after the brokerage report is generated.

GetBrokerReport

To get a brokerage report, you need to take the account id, start date and end date of the report, but no more than 31 days. We send a request to generate a report to the API in generate _broker_report_request , get a taskId in response. After that, using this taskId, we get data from get _broker_report_response.

Let’s start writing code

Method for getting the date, taking into account the subtraction from the current date

const getDateSubDay = (subDay = 5, start = true) => {     const date = new Date();     date.setUTCDate(date.getUTCDate() – subDay);       if (start) {         date.setUTCHours(0, 0, 0, 0);     } else {         date.setUTCHours(23, 59, 59, 999);     }       return date; };

Report generation request 

const brokerReport = await (sdk.operations.getBrokerReport)({         generateBrokerReportRequest: {             accountId,             from,             to,         }, });

Result:

• As a result of the first execution of the command, we get the taskId. 
• The report starts to be generated on the broker’s side. When it is ready is unknown, we wait and periodically pull the taskId in anticipation of the report.
• Why? Because if the report is not ready, it throws an error. If the report is not ready on the broker’s side, then this is an error in your code. Please process: 30058|INVALID_ARGUMENT|task not completed yet, please try again later

The code for waiting and receiving a report looks something like this.

const timer = async time => {     return new Promise(resolve => setTimeout(resolve, time)); }   const getBrokerResponseByTaskId = async (taskId, page = 0) => {     try {         return await (sdk.operations.getBrokerReport)({             getBrokerReportRequest: {                 taskId,                 page,             },         });     } catch (e) {         console.log(‘wait’, e);         await timer(10000);         return await getBrokerResponseByTaskId(taskId, page);     } };

Then the same magic happens. We stop our script, start it again, we don’t have a taskId. We execute the code with the taskId request, but we no longer get the taskId, but immediately the report. Magic! And everything would be fine if it was always like this. But in a month there will be no data at all. Useful :

• A bit of theory is outlined here and here .
• Putting the code together, the draft will look something like this.

https://github.com/pskucherov/tcsstat/tree/step3.1 https://github.com/pskucherov/tcsstat/compare/step3.1

• If someone comes across this, then welcome to the issue . After they repair this magic, it will lose its power and will be somehow different. But at the current moment (03/21/2023) it works just like that.

GetDividendsForeignIssuer

Someone might think that the method is similar to the previous one and you can use a single method in which you only change the name of the operations. But they didn’t guess!  The naming there is very different both in the methods and in the returned information. And the page count starts from 0, then from 1. In order not to get confused in all this, it’s easier to write two different methods. Which is strange, because the logic of work is the same. I spat for a long time when I tried to make one method and there was less code. There will be no examples here.

GetOperationsByCursor

My favorite of the three. Although not the most accurate, but the most adequate. We make a request from the beginning of creating an account to the maximum possible date (closing an account or the current one). We get the answer, take the cursor and re-request as long as there is data.  And the code is more concise than in the examples above.

const timer = async time => {     return new Promise(resolve => setTimeout(resolve, time)); }   const getOperationsByCursor = async (sdk, accountId, from, to, cursor = ”) => {     try {         const reqData = {             accountId,             from,             to,             limit: 1000,             state: sdk.OperationState.OPERATION_STATE_EXECUTED,             withoutCommissions: false,             withoutTrades: false,             withoutOvernights: false,             cursor,         };           return await sdk.operations.getOperationsByCursor(reqData);     } catch (e) {         await timer(60000);         return await getOperationsByCursor(sdk, accountId, from, to, cursor = ”);     } };

The draft to run is here: https://github.com/pskucherov/tcsstat/tree/step3.3 https://github.com/pskucherov/tcsstat/compare/step3.3 Now we are ready to add receiving operations to our application. If done correctly, then you need to get brokerage reports for the entire existence of the account. And for the missing data, those same T-3s, reload from operations. But this can be separated into a separate article. Of the main nuances that you will encounter is to glue operations and a brokerage report.

•  If today you received a brokerage report and transactions for the required dates, put it all in the database, then there are no problems. 
• You will have problems tomorrow when you receive the next portion of data from the report and operations and decide to synchronize them with the existing database. 
• A lot of nuances about mismatched or changing id after processing
• Then for the OTC market, the id’s don’t match at all.
•  As well as the nuances of synchronizing instruments, which again do not coincide, due to the peculiarities of the API. But that is another story.

Let’s add getting information about operations to our application. The main question will be where the data will be processed and stored.

•  If you do it for yourself, you will consume the same data from different devices. Then you need to process and store data on the server.
• If you have a lot of different data consumed by many different users, then you need to decide what is more important: the speed of the users or the saving of iron on your side. Whoever can afford an infinite amount of hardware counts everything on his server and makes it super fast for users, saving the user resources, such as battery and traffic, which is very important on phones.

In turn, counting in the browser is not the most optimal solution in principle. Therefore, what is not expensive, we consider it on our server. We leave the rest to the client. I really want to take and calculate the commission on the server. But here comes the nuance called “interactivity”. Let’s say you have thousands of operations and it takes five minutes to receive them. What will the user have at this time? Spinner? Progress? Infa about how much was uploaded? It is ideal to use “active waiting” when the user in the process could already see something. Here is the Result:

• Page loading
• All invoices are requested
• After that, all transactions with commissions for executed transactions are requested for all accounts. As data is received, it is rendered in the browser.

In order not to filter the data in the events each time, we pull our own event for each account. Like this:

socket.emit(‘sdk:getOperationsCommissionResult_’ + accountId, {                 items: data?.items,                 inProgress: Boolean(nextCursor), });

The draft to launch is here: https://github.com/pskucherov/tcsstat/tree/step3 https://github.com/pskucherov/tcsstat/compare/step2…step3 Moving on. It’s great that you’ve read this line! 

Calculation and output of information of interest

Depends on who needs what information. Therefore, I immediately tell you the main nuances that you will encounter.

Working with prices 

Everyone who works with finance knows that money transactions should only be performed with whole numbers. Due to the inaccuracy of values ​​after the decimal point and the cumulative error with a large number of operations. That is why all prices are presented in the following MoneyValue format

field	type	Description
currency	string	String ISO currency code
units	int64	Integer part of the sum, can be a negative number
nano	int32	Fractional part of the amount, can be a negative number

We process them separately, then bring them to the price value:

quotation.units + quotation.nano / 1e9

The cost of futures contracts

The price of futures is presented in points, when you have a currency future, you need to know the rate. And of course the price in points and the price step. When you calculate the profit from transactions, this can shoot, because. if you calculate the total amount by multiplying the price by the quantity. Here you need to be careful. For now, we’ll see how it goes. This applies to currency futures, in other places everything is ok with this.

OTC market

This market has a lot of peculiarities, so let’s study operations on it separately. When you start synchronizing operations, it will turn out that you need to bring figi / ticker to the same form in order to correctly match the instrument. When you start synchronizing this with the brokerage report, it will turn out that the tradeID of the same transaction has letters at the beginning in the transactions and they are not in the brokerage report. Therefore, they cannot be compared … ahem-ahem … by comparison! I matched the trade time, ticker and matching that one tradeId is contained in another. Right, I don’t know. Whoever encounters this and who cares about it, come to the issue or start a new one.

Mathematical operations on tools

It is impossible, without looking, to perform mathematical operations with the entire list. In order not to add warm to soft, we always check the currency and process only if we are sure that the currency matches, and the points are converted to the desired currency. Armed with knowledge about working with bank numbers, we will calculate the commission spent on each of the accounts. Like this: https://github.com/pskucherov/tcsstat/tree/step4 https://github.com/pskucherov/tcsstat/compare/step3…step4    

Microservice is ready!

https://github.com/pskucherov/tcsstat As a homework, you can check if the service works with a slow connection, when connections are broken, when the Internet is disconnected, when errors or expired limits on the part of the broker. 

Conclusions and plans for the future

• Learned about basic operations and working with the Invest API
• Time spent ~ 10 hours
• Difficulty level ~ junior+ / low middle 

If you continue to refine the microservice, you might end up with something like this

https://opexbot.info

  This is my development, for those who are too lazy to understand, run and count on their own. I plan to add analytics there at the request of users. If you liked the article, then subscribe to my telegram channel .