You're looking at a past competition year · Check out the latest Rails Rumble »

Launch Site Open Sesame - Alibaba automated price tracking with Tor

This is an entry in the 2015 Rails Rumble. After checking it out, we encourage you
to leave feedback and then take a look at the hundreds of other amazing entries.

V200_unnamed

Description

Filed in Aggregation.

Open Sesame is an exploration into scraping data from websites that don't want to be scraped. You can find almost everything on Alibaba, in LARGE quantities. Open Sesame is a tool that allows a user to simply enter the link to a product they want to track, and we will pull pricing information from that page every single day so that the user can track prices over time.

Bonus: Any time Alibaba redirects our scraper to their login page (they do this a lot), we re-run the same request through Tor, so that we can hit the Alibaba page from a different location in the world. We keep doing this until we get onto the page's data.

A user can track the history of each IP address that was used for their requests as well

Built By Team "Schwaughlin Bergexander"

Built With

Interested in seeing what was used to build this application? Resources may include things like open source libraries, third party APIs, and icons.

Tor, Rails, Bootstrap, jQuery, JS, Ngrok, Heroku, Github, Mechanize, HTTParty

Gem set:

gem 'sass-rails', '~> 5.0'
gem 'uglifier', '>= 1.3.0'
gem 'jquery-rails'
gem 'jbuilder', '~> 2.0'

gem 'mechanize'
gem 'htt...

Read More...

Tor, Rails, Bootstrap, jQuery, JS, Ngrok, Heroku, Github, Mechanize, HTTParty

Gem set:

gem 'sass-rails', '~> 5.0'
gem 'uglifier', '>= 1.3.0'
gem 'jquery-rails'
gem 'jbuilder', '~> 2.0'

gem 'mechanize'
gem 'httparty'

gem 'jquery-datatables-rails', github: "rweng/jquery-datatables-rails", branch: "master"
gem 'will_paginate'

gem 'unicorn'
gem 'thin'
gem 'geoip'

group :development do
gem 'web-console', '~> 2.0'
gem 'spring'
end

gem 'bootstrap-sass'
gem 'figaro'
gem 'omniauth'
gem 'omniauth-github'
gem 'pg'
gem 'pundit'
gem 'sendgrid'
gem 'simple_form'

group :development do
gem 'better_errors'
gem 'binding_of_caller', :platforms=>[:mri_20]
gem 'guard-bundler'
gem 'guard-rails'
gem 'guard-rspec'
gem 'rails_layout'
gem 'rb-fchange', :require=>false
gem 'rb-fsevent', :require=>false
gem 'rb-inotify', :require=>false
gem 'spring-commands-rspec'
end

group :development, :test do
gem 'byebug'
gem 'factory_girl_rails'
gem 'faker'
gem 'pry-rails'
gem 'pry-rescue'
gem 'rspec-rails'
gem 'rubocop'
end

group :production do
gem 'rails_12factor'
end

group :test do
gem 'capybara'
gem 'database_cleaner'
gem 'launchy'
gem 'selenium-webdriver'
end

### Testing gems
gem 'net'
gem 'socksify'
###

comments powered by Disqus