Project Overview
Look Estate is a real estate listings aggregator focused on pulling property data from multiple external sites, normalizing it into a consistent format, and presenting it through a searchable browser interface. The product spans Python scraping pipelines built with Scrapy, scheduled jobs that keep listings up to date, a FastAPI backend for serving and storing normalized data in PostgreSQL, and a Vue.js frontend with search, filtering, and pagination so users can browse aggregated inventory without visiting each source site individually.
Link to the project: Look Estate
Tech Stack
- Python (Scrapy, FastAPI)
- Vue.js
- PostgreSQL
Key Challenges
- Data Fragmentation: Data was spread across multiple websites, each with a unique HTML structure, making scraping and consolidation difficult.
- Data Consistency: Ensuring that the scraped data was clean, consistent, and free of duplicates or errors was crucial.
- User Experience: The frontend needed to be intuitive, responsive, and visually appealing to attract and retain users.
Solutions
Data Scraping with Python:
- Leveraged the Scrapy library to asynchronously scrape data from multiple sources. This included implementing data cleaning, validation, and pipelining.
- Set up a scheduler using cron jobs to periodically scrape data and update the database in real-time.
Backend Development:
- Built a backend API using FastAPI to serve the scraped data.
- Designed a database model for efficient storage and retrieval of listings.
Frontend Development with Vue.js:
- Created a responsive and interactive frontend using Vue.js to display listings in a user-friendly way.
- Implemented features such as search, filtering, and pagination to help users easily find what they were looking for.
- Utilized Vuex for state management to ensure a smooth and consistent user experience.
Lessons Learned
- Gained hands-on experience in UI/UX design using Figma, learning to create user-centric interfaces that enhance usability and engagement.
- Developed proficiency in frontend development with Vue.js, including state management, component-based architecture, and responsive design.
- Mastered building scalable data scraping pipelines, ensuring efficient data collection and processing from diverse sources.
- Learned the importance of flexibility and adaptability when scraping data from websites with varying HTML structures, which required creative problem-solving and robust error handling.
Attachments
Look Estate Home Page
Look Estate Detail Page
Link to the project: Look Estate