Day Trading with a Data Scientist Mindset: (1) Why Day Trade?

80% of day traders quit within two years. How can you decide if you’ll be in the other 20% and then go on to be a successful trader?

This article is the first in a series that looks at how we can apply the rigor of data science to day trading. In this first article, we formulate the day trading problem, setting goals and expectations for a successful day trading approach. Over the remainder of the series, we will look at data sources, formulate and evaluate approaches, call out many of the gotchas that lead people to abandon day trading, and finally implement some fully automated day traders which are profitable enough to beat the market.

At the end of 2023, I left my position as Data Scientist Architect at a successful community bank. I was frustrated that, even at an innovative bank — walking the cutting edge of Banking as a Service (BaaS) — the appetite for non-trivial data science approaches was very limited. I have since discovered this data conservatism to be common across community banks as an industry.

Disappointed to leave a job I had truly enjoyed, but hungry to exercise the data science muscles I had developed, I embarked on a holiday project in day trading, to see if I could devise a day trading scheme that utilized data science approaches to produce a reliable long-term profit. My initial trading approaches produced a modest profit, but I started the new year with confidence that I could implement something much better. I have been producing ever improving results over the subsequent months — and through this have also developed a methodology for evaluating and implementing potential trading schemes. In this series of articles I will share what I’ve done, in the hopes that it will encourage more data scientists to go from hypothetical trading hobbyists to successful day traders.

What Does Winning Look Like?

The definition of success will be different for each would-be trader, but should include all of these factors:

• Better profit than some default alternative. If I wasn’t day trading, my funds would be invested in an index mutual fund, tied to S&P 500. So, my successful trading scheme needs to perform better than that.
• Acceptable monthly income. I want my day trading scheme to produce enough income to cover my monthly expenses. That will enable me to pursue other interests, without worrying about paying the bills with those pursuits.
• Acceptable daily time commitment. My day trading scheme needs to be fully automated. On each trading day, the automation looks for trading opportunities, defines entries and exits, and executes the trading plan. My daily time commitment is to analyze the results and look for opportunities for improvement.

When I tell friends and family that I day trade, the initial response is almost always something like, “How can deal with the risk and uncertainty? Aren’t you worried about losing money?” I love to point out that the funds in a 401K fluctuate a great deal on a daily basis, and with day trading I can tune my algorithms to match my own risk tolerance. Let’s take a look.

Over the past 12 months, SPY (an exchange-traded fund (ETF) that is continually balanced to match the performance of the S&P 500) has increased 26% in value — for an average daily gain of 0.096% across 246 trading days. During that time:

• SPY has decreased in value on 42% of the days, losing at least 1% of its value on 7.6% of the days
• SPY has gained at least 1% of its value on 9.6% of the days and gained at least 2% on 0.4% of the days (one day)

The chart below shows the daily change in value for SPY over the past 12 months, and a histogram of the changes.

So, we can see that, though the market moves up through the year, it does so chaotically — on nearly one in five days, the market jumps up or down by 1%! And this is for a 12 month period of what has been perceived to be consistent growth. What if we look back 8 years?

• SPY has increased 168% during that time, for an average daily gain of a meager 0.057%
• SPY has decreased in value on 45% of the days, losing at least 1% on 11% of the days and losing at least 2% on 3.7% of the days
• On the worst day it lost nearly 11% of its value!
• SPY has gained at least 1% of its value on 13.5% of the days and gained at least 2% on 2.6% of the days
• On the best day it gained 9%

I use this data to define a baseline success for any trading scheme I am willing to use:

• It needs to gain an average of 0.1% per day
• It must never lose more than 2% on a day

Through the magic of leverage (for day trades, I can trade up to 4x my capital), a trading scheme gaining 0.1% per day, will produce 0.4% per day, which is sufficient to produce my needed monthly income. Nice!

Note: All the code needed to produce these charts is available on GitHub here: https://github.com/WolfgangGreen/TraderBlog001

Units of measure

If I’m averaging $500 per day at day trading, am I doing well? Obviously, it depends… If I’m investing $1,000,000, I’m gaining 0.05% per day and am just keeping up with the average pace of the S&P 500 over the past 8 years. If I’m investing $100,000, I’m gaining 0.5% per day and have basically found a printing press for money (more than tripling my capital every year).

Similarly, if a stock goes up $1.00, is that a big gain? For Booking Holding Inc (BKNG), that is a 0.25% change to the $4,000 stock price. But for Warner Brothers Discovery (WBD), that is a 14% change to the $7.24 stock price.

For all the analysis I do with investment gain and stock prices, I reason about percentage changes. Looking for strong surge in a stock price? Think about what percentage gain.

What is Day Trading?

This series looks at approaches I apply to day trading. Let’s take a quick look at common forms of trading. There are multiple taxonomies of traders, but this seems most common when discussing individuals who trade:

• Scalping: These trades typically last seconds — sometimes extending into a small number of minutes. The trader is looking for indicators predicting small price changes and gets in and out of the trade in a hurry. These traders are frequently competing with the high-frequency trading systems of the big investors, and that’s a battle I’m not looking to fight.
• Day trading: These trades typically last from several minutes to a couple hours. The trader is looking for larger gains that can be completed within a trading day. I consider this to be the sweet spot for my automated trading.
• Swing trading: These trades are typically held for several days and are looking for larger trends in stock pricing. The fees associated with leveraged trades lead me away from swing trading, but I’ll likely look at this later. Where Scalping and Day Trading typically don’t look at news relevant to a particular company, swing traders often consider this additional context, which is interesting from a data science and machine learning standpoint.
• Position trading (or simply Investing): These trades are held for days, months, to even years. People invest in individual stocks because they believe that the companies behind the stock will outperform the rest of the market.

All the approaches I talk about in these articles are in the Day Trading space and I’m typically looking for 3 to 5 trading opportunities during a trading day.

Searching for an Edge

Day trading is the repeated process of identifying a trading opportunity, defining the entry and exit points for the trade, and executing the trade plan. How do we identify good trading opportunities? There are myriad approaches used by traders, but all successful trading approaches need to have an edge: a reason to believe that this trading approach will not only be profitable over the long run, but that it will perform better than our default alternative.

As a data scientist, I want to use all the tools available to me to evaluate potential trading schemes before exercising them with actual money. This means a lot of simulation and a lot of analysis of these simulated results. I’m devoting an entire article to evaluating potential edges — and another one to simulating trade plans. I believe getting this right is the key to successful trading.

Roadmap for the other Articles

Thanks for reading. If you’re interested in reading more, please give me a Clap and subscribe: https://findahappy.medium.com/subscribe

Here is the what I’ll be covering:

1. Why Day Trade? (This article.) Setting goals and expectations for a successful day trading approach.
2. Data Sets: Sources and Test versus Training. We’ll look accessing stock data from Yahoo Finance and Alpaca.
3. Evaluating an Edge. How to identify potential trading edges and predict their profitability
4. Building a trading plan and simulating its performance
5. Building an actual trading harness
6. Gotchas to look out for. In simulating trading plans it is easy to make little mistakes that make an edge appear much more lucrative than it really is.
7. More Complicated Trading Schemes. Shorting and arbitrage
8. Machine Learning