Vic Yang

Python Crawler Example (1)

2017-07-28T10:00:00+00:00

I found clicking on job posting website could be automized, so I wrote a script for crawling. Here I used Python, beautifulsoup, requests and generators tricks for practicing.

I used Salesforce Career Page for example.

Salesforce Career Website

Where should I get started?

I knew Google Chrome has tool to see each request sent from browser, so I used that to see the request I sent from browser. By seeing that, I could know what kind of information I can wrap in code and issue a query.

Chrome Browser Developer Tool

Curl Example

I pasted on editor and saw there was a GET request sent to Salesforce server. So, all I need is use requests to send a GET request to that and I can get the result.

Apply this code would receive 403. (HTTP Code 403 - Forbidden)

What happened then?

Apparently, it’s not enough for this request, so I guessed it required header information to send it.

And I received 200. (HTTP Code 200 - OK)

Start analyzing HTML

I put the response in BeautifulSoup and got parsed HTML. What I really need is the list of jobs. I observed that page from Chrome Developer Tool and found some important CSS tag.

Result Example

From this code, I got only 10 jobs listed on my terminal. There must be something wrong, and then I realized I only parsed one page.

Next part, I would use generator to generate different url.

Generate URL by leveraging generators

I composed a get_total_pages function to get total pages. I found it’s only 10 jobs listed on one page, so I used the jobnumbers divided by 10 and got the result.

The function is the logic of generator.

def url_cat(total_pages, url):
	yield url
	page = 1
	while page <= total_pages:
    	yield url + str(page)
    	page += 1

I used 2 yield statements because the first url has no number concatenated afterward, and then I used the second yield to generate urls.

url_gen = url_cat(total_pages, url)

for url in url_gen:
    soup = get_soup(url, headers, payload)
    list_jobs(soup)

I applied for-loop to iterate the generator.

Now, I can get a list of information that I really wanted.

===

What’s inside?

Python
Generators
Requests
BeautifulSoup

Python Functions with some tricks

2017-07-25T10:00:00+00:00

Define functions is one of basic Python skills. I would like to focus on closure and its tricks.

Metadata

We can add some annotations on function in order to let programmers know how this function should be used.

Python interpreter wouldn’t give any semantic meaning on these statement. They are not type checks mechanism. However, they still give other maintainers enough clues to track code.

If functions have multiple values to return, they are packed in tuple and returned.

Default value in parameters

Indicated default values should always be immutable objects Like None, True, False, number or string

Capture variable in lambda

Using functools.partial

Closure

When do we have a closure?

We must have a nested function (function inside a function).
The nested function must refer to a value defined in the enclosing function.
The enclosing function must return the nested function.

When to use closures?

Closures can avoid the use of global values and provides some form of data hiding. It can also provide an object oriented solution to the problem.

We may want to replace single-method class with closure.

We may also want our callback functions to bring additional information

Reference

[1] Python Closures [2] Python cookbook 3rd version

Django with AJAX update

2017-07-24T10:00:00+00:00

When I was developing Django application, I need a feature that doesn’t require browser to refresh. Therefore, one way to achieve this feature is to use AJAX. When developing Django application, I think I wrote a sample code here for future reference.

Reference

How to work with AJAX request with Django

AJAX - Django

Matplotlib Introduction and Using

2017-07-24T09:00:00+00:00

This slides is made for Taipei.py meetup. In this talk, I gave base knowledge how to use Matplotlib and gave some exercises in order to let readers understand what and why they need to know this knowledge before applying it.

What is Matplotlib?

Matplotlib is a tool that implemented in Python. The primary target of this tool is another replacement of Matlab. Matplotlib has a convenient command-line interface. You can also trigger IPython to fast plot. You can use OO interface to compose complex figures.

Pros
- You can directly use Python to manipulate data and tweak.
- Multiplot is better in Matplotlib.
- You can apply LaTex to render plot.
Cons
- Because of the two interfaces (OO and GUI) exist, API is more complicated. You would see there are two or more methods to compose the same figures.
- Official document is hard to read.

Practice 1

Practice 2

Practice 3

Compare Between Shell and Jupyter

Summary

In Matplotlib, class is following the rule Figure -> Axes -> (Line2D, Text, etc). One Figure can contain several Axes. Use Axes in Matplotlib indicates a plotting area.

For more details, you can check my slides.

Matplotlib 簡介與使用 from Vic Yang

Login/Logout System with Django

2017-07-23T11:20:00+00:00

Summary

Just a simple demo for login/logout system with Email verification.

Also, deploy for Live Demo on Heroku

Prerequisites

Python3
PostgreSQL>=9.4
Mac/Linux

Use virtualenv

python -m venv ENV
source ENV/bin/activate

Clone this repository

git clone https://github.com/vrootic/i2x_challenge.git

Install packages

cd i2x_challenge
pip install -r requirements

Initialize DB

createdb i2x_challenge
python manage.py migrate

Create Superuser

python manage.py createsuperuser

Start Dev Server

python manage.py runserver

Deployment instructions

Install Heroku Toolbelt

First, sign up to Heroku Then, install Heroku toolbelt

heroku login

Create Heroku app

heroku create [your_app_name]

Set environment variables

heroku addons:create heroku-postgresql:hobby-dev
heroku config:set DJANGO_SECRET_KEY='[your secret key]'
heroku config:set DJANGO_SETTINGS_MODULE='config.settings.production'
heroku config:set DJANGO_ALLOWED_HOSTS='.herokuapp.com'

Push to deploy

git push heroku master

Migrate the DB

heroku run python manage.py migrate

Finish!

Link to live demo on Heroku

Live Demo on Heroku

RDSS Audit System

2017-07-23T10:20:00+00:00

RDSS Audit System

This project is when I was serving in Alternative Military Service in Taiwan.

The project accelerated my daily tasks and brought benefits to accounting room.

This project has several goody features including,

Registration searching with part of information(names, ID, or echelon)
Calculate the salary for the next month
Record the base salary and automatically store records
Categorize in service/out of service people
Upload excel file from other department and audit those records

Screenshot 1

Screenshot 2

Screenshot 3

–

What has inside?

Django
JavaScript
Bootstrap

–

File Compare with ReactJS

2017-07-23T10:10:00+00:00

File Compare with ReactJS

Summary: A simple file compare tool with ReactJS

FileCompare

Compare two Excel files

What can it do for us?

Compare two Excel file (.xls or .xlsx) by selecting common attributes. If there are some different rows or duplicate rows, this program will show them for you.
The different rows can be exported as .csv file.

How does the comparing process take action?

In the beginning, we preselect the “authentication ID” as the primary field, and the program would check whether the given fields are the same based on the primary field.
In the next version, the program will choose which field can be the primary field.

How can we use the program?

Install Electron http://electron.atom.io
Clone this respository to local machine

git clone https://github.com/vrootic/FileCompare.git

Execute it.

electron FileCompare

Screenshot 1

Screenshot 2

–

What has inside?

Electron
ReactJS
Bootstrap

–

Check it out here

Bane - A simple calculator

2017-07-23T09:20:00+00:00

Bane

A simple calculator for internal use.

The simple program would calculate the base number multiplied by a propotion.
For example, we are going to pay someone, and we want to pay them according to the proportion of their workdays in this month.
There are other use cases.
It will show abbreviation of month which acrossed the date range if the date range is over two months.

Installation

The project is based on Qt4, Python 2.7, and PySide. It was only tested on Mac.

brew install qt
pip install pyside
pyside_postinstall.py -install

Check it out here.

Mac Show Hidden Files

2017-07-22T23:10:00+00:00

Just found some tricks to show/hide Mac files

defaults write com.apple.finder AppleShowAllFiles YES

alt + right click on Finder ==> Relaunch

defaults write com.apple.finder AppleShowAllFiles NO

alt + right click on Finder ==> Relaunch

CSS Triangle

2017-07-22T23:00:00+00:00

I saw a blog which talked about how to make a css triangle. Through simple box model, we could build css triangle.

Here is my note.

.css-initial-box {
  width: 100px;
  height: 100px;
  padding: 16px;
  border: 12px solid transparent;
  border-top-color: red;
  border-right-color: green;
  border-bottom-color: blue;
  border-left-color: purple;
}

.css-square {
  width: 0px;
  height: 0px;
  border: 112px solid transparent;
  border-top-color: red;
  border-right-color: green;
  border-bottom-color: blue;
  border-left-color: purple;  
}

.css-bottom-triangle {
  width: 0px
  height: 0px;
  border-bottom: 112px solid blue;
  border-left: 112px solid transparent;
  border-right: 112px solid transparent;
}