Agriculture census (tehsil level)

[kapildadheech]

Ticket Contents

Scraping of data from the Agri Census website, publish as vector asset, use in GEE pipeline to get a tehsil vectorized map. Especially relevant since crop classification is a hard problem

Goals

Goals

• Scraping of data from the Agri Census website,
• publish as vector asset, use in GEE pipeline to get a tehsil vectorized map. Especially relevant since crop classification is a hard problem

Expected Outcome

Extent of different crop types (tehsil)

Acceptance Criteria

Data Scraping

• Given the Agri Census website is accessible, when the scraper runs, then all relevant tehsil-level boundary and attribute data should be extracted without missing fields.

• Given the website has paginated or tabular content, when the scraper is executed, then it must handle pagination and table parsing correctly.

• Scraped data should be stored in a structured format (GeoJSON/CSV with geometry).

Vector Asset Publishing

• Given the scraped tehsil-level data, when the transformation script runs, then the data must be cleaned (valid geometries, proper CRS, consistent attributes).

• The processed data must be published as a vector asset (GeoJSON or shapefile uploaded to Earth Engine).

• Metadata (source, date, schema) must be attached to the asset.

GEE Pipeline Integration

• Given the vector asset is available in Earth Engine, when the GEE pipeline runs, then it must use the asset to generate a tehsil-level vectorized map.

• The pipeline should confirm that tehsil boundaries align with existing project geometries.

• The output should be validated by visual overlay on GEE to ensure no missing or misaligned tehsils.

Implementation Details

• Python notebook or integrate directly into computing module

Mockups/Wireframes

No response

Product Name

Agriculture census (tehsil level)

Organisation Name

C4GT

Domain

No response

Tech Skills Needed

Python

Organizational Mentor

@amanodt @kapildadheech @ankit-work7

Angel Mentor

No response

Complexity

Medium

Category

Backend

[ShivamBhoyar]

Hello Sir, I would like to work on this issue and will use my technical skills to resolve it. However, first, could you please provide me with mockups or a rough sketch to help me understand the project more clearly?

[aaditeshwar]

Thanks @ShivamBhoyar for the interest. Sharing some pointers:

• Check out the agri-census website https://agcensus.da.gov.in/DatabaseHome.aspx and also related websites like https://upag.gov.in/ which publish tehsil and district-level stats.
• Build a list of properties you would like to associate with each tehsil or district, E.g.: Crop-1, Area under crop-1, Crop-2, Area under crop-2, etc.
• Study the data to check if it is consistent - no spelling variations in crop names, consistent units of acres or hectares for the area.
• Once satisfied, write a scraper to pull the data and store in a CSV.
• Join the CSV with CoRE stack tehsil and district maps. These are available on the CoRE stack pan-India GEE app. The joining will have to be done based on string matches between the tehsil name in the agri-census data and the tehsil name in the CoRE stack data, since common IDs are not being used on the agri-census and other websites. This entity resolution can be done through edit distance and phonetic distance matches. But it will not be 100% accurate or complete, so some sample check of matches done, and manual matching of unmatched tehsils, will need to be done.
• Once ready, this data can be converted to a map and published. Further pipelines can be written to extract this into tehsil-level reports that are published on apps and dashboards like Know Your Landscape.
• Sample outputs can be shared and validated.

@kapildadheech , @amanodt , @ankit-work7 , @nirzaree-cfpt , FYI.

[ShivamBhoyar]

Sir, I have some doubts and need to confirm the CSV file format. Could you please tell me how I should contact you, or should I share my doubts here itself?

[aaditeshwar]

Sure, please feel free to drop me an email at aseth@cse.iitd.ac.in

[aaditeshwar]

Hi all, Do please join the discord channel and googlegroup linked from this announcement of the CoRE stack innovation challenge, and do participate in the challenge too: https://core-stack.org/core-stack-innovation-challenge-1st-edition/. Our first community call on Friday 3-4pm will be very useful to get started.

[ShivamBhoyar]

I have just sent you the email detailing my progress, including the proposed Demo CSV file structure.

[neha222222]

hi @aaditeshwar, put up a PR here - #815

got the scraper and tehsil matching working. will run it on a few states and share the outputs soon.