A Major Step in Stock Analyzer

[SrigadaAkshayKumar]

New updates in project

[SrigadaAkshayKumar]

We are currently implementing a major update in our project. Previously, we were continuously fetching data directly from the YFinance API. However, this approach led to issues such as rate limiting and execution timeouts due to the large volume of requests.

To resolve this, we’ve decided to integrate an ETL scheduler using Apache Airflow. This scheduler will automate the process of fetching stock data once a day for a predefined list of stocks and store it in a database.

Since the project is still in the development and testing phase, we are currently storing the data in a CSV file (with a limited number of stock entries). When a user tries to request data for a stock that is not in our list, the system will display a message like:
“The selected stock is not yet Listed in Stock Analyzer. Please try another listed stock.”

When the project is migrated from Render to Google Cloud Platform (GCP), we will replace the CSV file with a MySQL database for better scalability and performance.

How it works:

Backend requests no longer fetch real-time data from YFinance.

Instead, they read the latest available data from the CSV (or database).

The Apache Airflow scheduler fetches only one day’s worth of new data (period=1d) daily and appends it to the existing data — avoiding the need to reload the full dataset each time.

This architecture significantly improves reliability and efficiency, especially for long-term forecasting, and keeps API usage within safe limits.

[SrigadaAkshayKumar]

We welcome all backend developers to share your suggestions and improvements regarding this major update. The implementation work will begin on 1st August.

Anyone interested in contributing is most welcome

Special Note for GSSoC Contributors:
Contributors under GSSoC who contribute to this update will receive an “Advanced” tag on their PRs, which carries higher points in GSSoC scoring, along with bonus merit points for their efforts.

We’re excited to collaborate and build this together

[Ritikavarshney-hub]

kindly assign this issue to me

[SrigadaAkshayKumar]

@Ritikavarshney-hub, I appreciate your interest in being part of the project. Are you comfortable with updating the project? Please let us know which part you'd like to work on and if there’s any area where you need help, so we can divide the project into modules accordingly

[SrigadaAkshayKumar]

sample workflow

[Ritikavarshney-hub]

Hello sir, I wanted to work on the issue Upgrading stock analyzer with LSTM on keras for better accuracy

…

On Thu, 31 Jul, 2025, 9:00 pm Srigada Akshay Kumar, < ***@***.***> wrote: @Ritikavarshney-hub <https://github.com/Ritikavarshney-hub>, I appreciate your interest in being part of the project. Are you comfortable with updating the project? Please let us know which part you'd like to work on and if there’s any area where you need help, so we can divide the project into modules accordingly — Reply to this email directly, view it on GitHub <#61 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BL22Q4K2N35NCMTVO4FZHKL3LIZBTAVCNFSM6AAAAACCW24WXWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTGOJUHEYDOOA> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[SrigadaAkshayKumar]

proposed backend directory structure

Here is the proposed backend structure currently we have combined all api's in app.py but now we are using modular approach so we divide the services as
│ │ └── stock_service.py # Read/process CSV data sends graph and table data to frontend also contains news logic
│ │ └── stock_predict.py # contains prediction graph and prediction logic

so that we can easily complete the tasks one by one

backend/
│
├── app/                            # Main Flask application package
│   ├── routes/                     # Flask route handlers
│   │   └── stock_routes.py     # API: GET /stock/<symbol>, etc.
│   ├── services/                   # Data logic layer
│   │   └── stock_service.py        # Read/process CSV data sends graph and table data to frontend also contains news logic
│   │   └── stock_predict.py       # contains prediction graph and prediction logic
│   └── config.py                   # App configuration (CSV paths, env)
│
├── data/                           # Daily updated CSVs by Airflow DAGs
│   └── stock_TCS.csv
│   └── stock_ITC.csv
│   └── ...
│
├── airflow_dags/                   # Apache Airflow DAGs
│   └── fetch_stock_data_dag.py     # Daily fetch and append logic using yfinance
│
├── Dockerfile                      # Dockerfile to containerize Flask app
├── requirements.txt                # Python dependencies (Flask, pandas, etc.)
├── .env                            # Environment variables (used in config.py)
├── app.py                          # Entry point for Flask (runs the app)

any suggestions or improvements?