Headless Selenium on Ubuntu 14.04 with PhantomJS 2.0

Headless Selenium on Ubuntu 14.04 with PhantomJS 2.0

N.B. This is STILL incomplete.

More work later :)

We’re following the excellent Headless Selenium Testing With Python and PhantomJS

Install Selenium

Let’s use pip to install Selenium

$ pip install selenium

Install PhantomJS

To make PhantomJS installation easy we will use the GitHub Repository, PhantomJS 2.0 built on Ubuntu 14.04 x64.

  1. Let’s login as root, cd to the root home.
    $ # However you login as root - I use WinSCP & Putty.
    $ cd ~
    
  2. According to the notes on the GitHub, to avoid the noticed dependency error, first let’s install “libicu52”.
    $ sudo apt-get install libicu52
    
  3. Now let’s just follow the original instructions on the GitHub Repository.
    $ # Since the file the GitHub Repository is 
    $ #    **actually** the phantomjs **executable**, 
    $ # And we want system wide access,
    $ cd /usr/bin
    
    # Now do our GitHub Repository wget into /usr/bin
    $ $ wget https://github.com/Pyppe/phantomjs2.0-ubuntu14.04x64/raw/master/bin/phantomjs
    
    # Now change the permissions
    $ chmod 0755 phantomjs
    
    $ # -rwxr-xr-x 1 root root 56596085 Aug 26 09:22 phantomjs
    

    That does it.

Love and peace,
Joe

Download an App from Google Play to a PC & Install it on a Tablet (Sideload)

Download an App from Google Play to a PC & Install it on a Tablet

Download App to PC

ref How to Download Android Apps from Play Store to PC?

  1. Find the App URL on google play store for your account
  2. Here’s the App URL for Cogi – Notes & Voice Recorder
    https://play.google.com/store/apps/details?id=com.cogi.mobile
  3. Copy the id == com.cogi.mobile
    From the App URL
    To https://apps.evozi.com/apk-downloader/
  4. Click on newly appeared button
    “Generate Download Link”
Result:
  Package Name: com.cogi.mobile [Play Store]
  File Size: 12.1 MB
  QR Code: View
  MD5 File Hash: ee75eba0fef749dba8796c2eb5cab309
  Last Fetched: 2015-08-03 21:31:36
  Version: 1.17.2 (94)
  1. Click on the newly appeared button
    “Click here to download =com.cogi.mobile now”
  2. Put the download into yourPC PATH_TO_Apps/Cogi/
  3. Verify the md5 hash
>cd yourPC PATH_TO_Apps/Cogi/
>rem use whatever md 5 checker you have
>md5chk.py com.cogi.mobile.apk ee75eba0fef749dba8796c2eb5cab309
      file length: 12736720
    file checksum: ee75eba0fef749dba8796c2eb5cab309
expected checksum: ee75eba0fef749dba8796c2eb5cab309
True

Install the Downloaded APK on the Tablet

refs:

  1. Connect USB cable from tablet to pc
  2. Copy [drag & drop] com.cogi.mobile.apk
    From: yourPC PATH_TO_Apps/Cogi/
    Into: yourPC PATH_TO_Tablet/Tablet/Download
    N.B.: Navigate to yourPC PATH_TO_Tablet/Tablet/Download in your PC File Browser
    when your Tablet is USB connected.
  3. Disconnect USB cable from tablet to pc
  4. On Tablet Go to Menu > Settings > [General] > Security > and check “Unknown Sources”
    4.1 the cnet link ALSO says
    Menu > Settings > [General] > Security > and check “Verify Apps”
    “Depending on your device, you can also choose to be warned before installing harmful apps. This can be enabled by selecting the Verify apps option in the Security settings.” N.B. For me this was ALREADY CHECKED
  5. On Tablet go to My Files > Download
    N.B. “Downloaded Apps” is a Category NOT A FOLDER
  6. Tap com.cogi.mobile.apk
    [I notice that this app gets A TON of permissions – should have checked earlier]
  7. Tap Install
    RESULTS:
    – Cogi appears in “Downloaded Apps” Category
    – Cogi appears in the Apps Tab of the Top Screen
  8. IMPORTANT Go back to Menu > Settings > [General] > Security > and UNCHECK “Unknown Sources”

THE END

Uninstalling the Bing Bar, Surreptitiously Installed with Skype 5.8

Uninstalling the Bing Bar, Installed with Skype 5.8

Background

I got a Windows error message generated by BSvcProcessor.exe. I didn’t install this and so i did some research.

Research Links

This link contained “BSvcProcessor.exe”; it hooked up “BSvcProcessor.exe” with Bing Service, [Tom’s Hardware microsoft-bing-service…] (http://www.tomshardware.com/answers/id-2735458/microsoft-bing-service-bingsvc-exe.html)

Persuit of “Bing Service” searches produced “What is the Microsoft Bing Bar” from support.skype.com. This link said:

The Bing Bar is installed automatically when you install Skype 5.8 for Windows desktop.

Uninstall

So I uninstalled it.

To uninstall the Bing Bar:

Click Start - Control Panel.
Windows XP: Add or Remove Programs.
Windows Vista and 7: Uninstall a program.
Select Bing and click Uninstall.

Ubuntu 12.04 Postgres Access for C

Ubuntu 12.04 Postgres Access for C

Description

This post describes the steps to install the software needed:

  • to be able to access a PostgreSQL database table from C,
  • starting from a DigitalOcean Ubuntu 12.04 Droplet, Virgin Image.

Step 1 Create DigitalOcean Ubuntu 12.04 Droplet

  • Log into DigitalOcean (DO) Droplets
  • Click the “Create Droplet” Button at the top right.
  • Name your Droplet Hostname. Select Size. Select Region.
  • Select Image, On “Distributions Tab”/UbuntuTile_bottom_half, click & select “12.04.05 x32”
  • Click Big Green Button at bottom “Create Droplet”
  • Copy the Droplet ip address for later access.

Step 2 Install PostgreSQL on Ubuntu 12.04

I follow “How To Install and Use PostgreSQL on Ubuntu 12.04”.

root@whatever:~# # logged in as root
root@whatever:~#
root@whatever:~# apt-get update

[**DELETED LINES**]

Reading package lists... Done

root@whatever:~#
root@whatever:~# # download Postgres and its helpful accompanying dependencies
root@whatever:~# sudo apt-get install postgresql postgresql-contrib

Reading package lists... 0%

[**DELETED LINES**]

Moving configuration file /var/lib/postgresql/9.1/main/pg_ident.conf to /etc/postgresql/9.1/main...
Configuring postgresql.conf to use port 5432...
update-alternatives: using /usr/share/postgresql/9.1/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster.1.gz (postmaster.1.gz) in auto mode.
* Starting PostgreSQL 9.1 database server
[ OK ]
Setting up postgresql (9.1+129ubuntu1) ...
Setting up postgresql-contrib-9.1 (9.1.15-0ubuntu0.12.04) ...
Setting up postgresql-contrib (9.1+129ubuntu1) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place

Step 3 Add a new [Linux] User

The "How To Install and Use PostgreSQL on Ubuntu 12.04" article says the next step is to "Create Your PostgreSQL Roles and Databases".

However, first, before that, I will add a new [non root, Linux] User, since this is a virgin image and I don't want to be messing around in root all the time.

I will however, give the new user Admin Privileges.

root@whatever:~# # Joe's first step is to add a new [Linux] user.
root@whatever:~# #     now following https://www.digitalocean.com/community/tutorials/how-to-add-delete-and-grant-sudo- privileges-to-users-on-a-debian-vps
root@whatever:~# #         assume you are currently logged in as the root user, pwd == ~
root@whatever:~#
root@whatever:~# adduser joed   # or whatever username you want

[**DELETED LINES** you get asked a lot of questions in here]

Is the information correct? [Y/n]
root@whatever:~# su - joed   # www.linfo.org/su.html says "The su (short for substitute user)..."
joed@whatever:~$ pwd
/home/joed
joed@whatever:~$ exit  # exit from joed goes back to root
logout
root@whatever:~# pwd
/root

Step 4 Grant [Linux] Users Administrative Privileges

I want the newly added Linux user, "joed", to have Administrative Privileges.

root@whatever:~#
root@whatever:~# # Grant Users Administrative Privileges
root@whatever:~# #    enable sudo privileges
root@whatever:~# #        run the   visudo   command
root@whatever:~# #            i set visudo command editor to be nano  - i hate vi
root@whatever:~# # see http://superuser.com/a/821888/236556
root@whatever:~# # using WinSCP & np++ i edited /etc/sudoers  DIRECTLY, adding the following lines under the Defaults section at the beginning
root@whatever:~# #    Defaults        editor="/usr/bin/nano"
root@whatever:~#
root@whatever:~#
root@whatever:~# # now using visudo [with nano] i add the following line
root@whatever:~# #    joed    ALL=(ALL:ALL) ALL
root@whatever:~# #    under these 2 lines
root@whatever:~# #    # User privilege specification
root@whatever:~# #    root    ALL=(ALL:ALL) ALL
root@whatever:~#
root@whatever:~# #    N.B. i used the visudo command instead of editing it directly with np++
root@whatever:~# #        because i read SOMEPLACE?? that visudo checks the syntax of the lines for correctness
root@whatever:~#
root@whatever:~#
root@whatever:~#
root@whatever:~# # now let's check to see if joed has sudo priveleges
root@whatever:~#
root@whatever:~# su joed
joed@whatever:/root$
joed@whatever:/root$
joed@whatever:/root$ # check if i have sudo priveleges
joed@whatever:/root$     # see http://superuser.com/a/553939/236556
joed@whatever:/root$
joed@whatever:/root$ sudo -v
[sudo] password for joed:
joed@whatever:/root$ sudo -l
Matching Defaults entries for joed on this host:
env_reset, secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin, editor=/usr/bin/nano

User joed may run the following commands on this host:
(ALL : ALL) ALL
joed@whatever:/root$ exit
exit
root@whatever:~#
root@whatever:~#
root@whatever:~# # OK joed is a new Linux user  &  has sudo priveleges.
root@whatever:~#

Step 5 Make the new Linux User a PostgreSQL "superuser".

Now we "pop the stack" back into "How To Install and Use PostgreSQL on Ubuntu 12.04" at the "Create Your PostgreSQL Roles and Databases" step.

Remember, in the big picture, we are making the Linux user, joed, become a PostgreSQL superuser & accessing a PostgreSQL table from the C language.

root@whatever:~#
root@whatever:~#
root@whatever:~# # Create Your PostgreSQL Roles and Databases
root@whatever:~# #    To begin creating custom users, first switch into the [PostgreSQL] default [Linux] user [== "postgres"]
root@whatever:~# sudo su - postgres      # i BELIEVE i don't need sudo because i am doing this as root but WHY NOT FOLLOW THE DIRECTIONS  :)
postgres@whatever:/root$
postgres@whatever:/root$ cd ~
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$ # OK so we are in the Linux user account named postgres, added by the PostgreSQL install
postgres@whatever:~$ #     now we will do a PostgreSQL createuser command  to make the existing Linux user, joed, be a PostgreSQL superuser
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$ pwd
/var/lib/postgresql
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$ createuser --pwprompt
Enter name of role to add: joed
Enter password for new role:  <enter the pw>
Enter it again: <enter the pw>
Shall the new role be a superuser? (y/n) y
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$ # now let's look for joed in the pg database
postgres@whatever:~$
postgres@whatever:~$ psql
psql (9.1.15)
Type "help" for help.

postgres=# \dt
No relations found.
postgres=# \l
List of databases
Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
|          |          |             |             | postgres=CTc/postgres
template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
|          |          |             |             | postgres=CTc/postgres
(3 rows)

postgres=# \du-- show users
postgres=# \du
List of roles
Role name |                   Attributes                   | Member of
-----------+------------------------------------------------+-----------
joed      | Superuser, Create role, Create DB, Replication | {}
postgres  | Superuser, Create role, Create DB, Replication | {}

postgres=# -- N.B. -- is the psql  COMMENT indicator
postgres=# -- list databases & tables for current db  http://dba.stackexchange.com/a/1288
postgres=# --     also above tells how to switch the currnent db
postgres=# -- OK logout of the psql interpreter (\q) & exit back to root Linux user & log back in as postgres_user joed Linux
postgres=# \q
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$ exit
exit
root@whatever:~#
root@whatever:~# # OK joed is a PostgreSQL superuser

Step 6 Create a PostgreSQL Database

I am going to create a PostgreSQL database named "omnia". I will do this as the PostgreSQL user named "postgres".

N.B We are back into "How To Install and Use PostgreSQL on Ubuntu 12.04" at the "Create Your PostgreSQL Roles and [now] Databases" step.

root@whatever:~#
root@whatever:~# # ok [from root] log back in as Linux user == postgres
root@whatever:~# su postgres
postgres@whatever:/root$ cd ~
postgres@whatever:~$ # re-follow https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-12-04
postgres@whatever:~$ # create your first usable postgres database
postgres@whatever:~$ #    i create a named "omnia"
postgres@whatever:~$ createdb omnia
postgres@whatever:~$
postgres@whatever:~$
postgres@whatever:~$ psql
psql (9.1.15)
Type "help" for help.

postgres=# \l
List of databases
Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
omnia     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
|          |          |             |             | postgres=CTc/postgres
template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
|          |          |             |             | postgres=CTc/postgres
(4 rows)

postgres=# -- -- http://stackoverflow.com/questions/3949876/how-to-switch-databases-in-psql
postgres=# \connect omnia
You are now connected to database "omnia" as user "postgres".
omnia=# \q
postgres@whatever:/root$ # let's try this as joed
postgres@whatever:/root$ exit
exit
root@whatever:~# su joed
joed@whatever:/root$ cd ~
joed@whatever:~$ pwd
/home/joed
joed@whatever:~$ # follow https://www.digitalocean.com/community/tutorials/how-to-install-and-use-postgresql-on-ubuntu-12-04
joed@whatever:~$     # log into the correct database==omnia(using the psql -d omnia)
joed@whatever:~$ psql -d omnia
psql (9.1.15)
Type "help" for help.

omnia=# -- YAHOO we're in as joed a postgres superuser
omnia=# \l
List of databases
Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
omnia     | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |
template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
|          |          |             |             | postgres=CTc/postgres
template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +
|          |          |             |             | postgres=CTc/postgres
(4 rows)

omnia=# \du
List of roles
Role name |                   Attributes                   | Member of
-----------+------------------------------------------------+-----------
joed      | Superuser, Create role, Create DB, Replication | {}
postgres  | Superuser, Create role, Create DB, Replication | {}

omnia=# \q
joed@whatever:~$
joed@whatever:~$ pwd
/home/joed

Step 7 Create & Initialize a Table

I will create the table named "test.

I will initialize (load it up) from a .csv file.

I will do this as the [Linux & PostgreSQL] user named "joed".

[joed@whatever trial_test_matrix]$ # now let's make the table named "test" in the "omnia" db
[joed@whatever trial_test_matrix]$ psql -d omnia
psql (9.1.15)
Type "help" for help.

omnia=# CREATE TABLE test(
omnia(#     id INTEGER PRIMARY KEY,
omnia(#     test_ix INTEGER,
omnia(#     origin CHAR(512),
omnia(#     max_prime INTEGER,
omnia(#     primes_tail_0 INTEGER,
omnia(#     primes_tail_1 INTEGER,
omnia(#     primes_tail_2 INTEGER,
omnia(#     primes_tail_3 INTEGER,
omnia(#     primes_tail_4 INTEGER,
omnia(#     primes_tail_5 INTEGER,
omnia(#     primes_tail_6 INTEGER,
omnia(#     primes_tail_7 INTEGER
omnia(# );
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "test_pkey" for table "test"
CREATE TABLE
omnia=# \dt
List of relations
Schema | Name | Type  | Owner
--------+------+-------+-------
public | test | table | joed
(1 row)

omnia=# -- show the column names for test
omnia=# \d test

Table "public.test"
Column     |      Type      | Modifiers
---------------+----------------+-----------
id            | integer        | not null
test_ix       | integer        |
origin        | character(512) |
max_prime     | integer        |
primes_tail_0 | integer        |
primes_tail_1 | integer        |
primes_tail_2 | integer        |
primes_tail_3 | integer        |
primes_tail_4 | integer        |
primes_tail_5 | integer        |
primes_tail_6 | integer        |
primes_tail_7 | integer        |
Indexes:
"test_pkey" PRIMARY KEY, btree (id)

omnia=# \q
[joed@whatever trial_test_matrix]$ # now load up the omnia.test table from init.dbcsv []n]o] ]h]e]a]d]e]r]s]
[joed@whatever trial_test_matrix]$
[joed@whatever trial_test_matrix]$ psql -d omnia
psql (9.1.15)
Type "help" for help.

omnia=# -- now import from csv
omnia=# --    see http://stackoverflow.com/a/2987451/601770
omnia=# --    we will do something like COPY zip_codes FROM '/path/to/csv/ZIP_CODES.txt' DELIMITER ',' CSV;
omnia=# --    COPY zip_codes FROM '/path/to/csv/ZIP_CODES.txt' DELIMITER ',' CSV;test/home/joed/expr/trial_test_matrix/init.csv
omnia=# -- load up the table from init.csv file NO HEADER, 1000 records
omnia=# COPY test FROM '<path to>/init.csv' DELIMITER ',' CSV;
COPY 1000
omnia=# -- yahoo
omnia=#
omnia=# SELECT test_ix, max_prime FROM test WHERE id<20;

test_ix | max_prime
---------+-----------
0 |       840
1 |       980
2 |       900
3 |       900
4 |       830
5 |       950
6 |       990
7 |       800
8 |       960
9 |       820
10 |       910
11 |       930
12 |       920
13 |       950
14 |       960
15 |       820
16 |       830
17 |       880
18 |       830
(19 rows)

omnia=# \q

[joed@whatever trial_test_matrix]$ # OK we have created a table named test
[joed@whatever trial_test_matrix]$ #      AND loaded it up with data from a .csv file

Step 8 Install gcc (the C compiller)

Aparently, gcc does not come with the standard DO Ubuntu image. Who'd 'a thunk it? :)

I'll install it as root.

[joed@whatever SQL]$
[joed@whatever SQL]$ # The tale of "woe" begins here. :)
[joed@whatever SQL]$
[joed@whatever SQL]$ # Try to compile my C program named tse_analogous_db.c
[joed@whatever SQL]$ #    located in the   .../SQL dir
[joed@whatever SQL]$
[joed@whatever SQL]$ gcc -c -I/usr/local/pgsql/include  tse_analogous_db.c
The program 'gcc' can be found in the following packages:
* gcc
* pentium-builder
Ask your administrator to install one of them
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$ # well that does that   -- i have to install gcc
[joed@whatever SQL]$ # see http://askubuntu.com/a/271561/144883
[joed@whatever SQL]$ #     via the toolchain PPA
[joed@whatever SQL]$ #          add the PPA to your system
[joed@whatever SQL]$ # i am going to do this as root
[joed@whatever SQL]$
[joed@whatever SQL]$ exit
root@whatever:~#
root@whatever:~# # OK now let's install gcc according to  http://askubuntu.com/a/271561/144883  - REALLY
root@whatever:~# #    via the toolchain PPA
root@whatever:~# #          add the PPA to your system
root@whatever:~#
root@whatever:~# # First command listed by the link i am following:
root@whatever:~# #    http://askubuntu.com/a/271561/144883
root@whatever:~# #        apt-get install python-software-properties
root@whatever:~#
root@whatever:~# apt-get install python-software-properties

Reading package lists... 0%
[**DELETED LINES**]
After this operation, 651 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
[**DELETED LINES**]
Setting up python-software-properties (0.82.7.7) ...
root@whatever:~#
root@whatever:~#
root@whatever:~# # Next command listed by the link i am following:
root@whatever:~# #    http://askubuntu.com/a/271561/144883
root@whatever:~# #        add-apt-repository ppa:ubuntu-toolchain-r/test
root@whatever:~#
root@whatever:~#
root@whatever:~# add-apt-repository ppa:ubuntu-toolchain-r/test
You are about to add the following PPA to your system:
Toolchain test builds; see https://wiki.ubuntu.com/ToolChain

More info: https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpNRzYg4/secring.gpg' created
[**DELETED LINES**]
gpg:               imported: 1  (RSA: 1)
OK
root@whatever:~#
root@whatever:~#
root@whatever:~#
root@whatever:~# # Next command listed by the link i am following:
root@whatever:~# #    http://askubuntu.com/a/271561/144883
root@whatever:~# #        apt-get update
root@whatever:~#
root@whatever:~# apt-get update


0% [Working]
[**DELETED LINES**]
Reading package lists... Done

root@whatever:~#
root@whatever:~#
root@whatever:~#
root@whatever:~#
root@whatever:~# # Next command listed by the link i am following:
root@whatever:~# #    http://askubuntu.com/a/271561/144883
root@whatever:~# #        apt-get install gcc-4.8
root@whatever:~#
root@whatever:~# #    The above link ALSO says
root@whatever:~# #        The latest available version of gcc, for 12.04, is 4.8.1 and is available in the toolchain PPA
root@whatever:~#
root@whatever:~# apt-get install gcc-4.8.1

Reading package lists... 0%
[**DELETED LINES**]
After this operation, 70.6 MB of additional disk space will be used.
Do you want to continue [Y/n]? y
[**DELETED LINES**]
Setting up manpages-dev (3.35-0.1ubuntu1) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
root@whatever:~#
root@whatever:~#
root@whatever:~#
root@whatever:~# # Next is the FINAL command listed by the link i am following:
root@whatever:~# #    http://askubuntu.com/a/271561/144883
root@whatever:~# #        update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
root@whatever:~#
root@whatever:~# update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-4.8 50
root@whatever:~#
root@whatever:~#
root@whatever:~# # OK gcc is now installed :)
root@whatever:~#

Step 9 Install libpq-dev to access PostgreSQL from C

The libpq-dev code library for accessing PostgreSQL from C needs to be installed.

I'll install it as root.

root@whatever:~#
root@whatever:~# su joed
joed@whatever:/root$ # shorten the prompt
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$ # install libpq-dev
[joed@whatever SQL]$ # Following http://askubuntu.com/a/182062/144883
[joed@whatever SQL]$ # Do this as root
[joed@whatever SQL]$ exit
exit
[root@whatever ~]#
[root@whatever ~]# apt-get install libpq-dev

Reading package lists... 0%
[**DELETED LINES**]
After this operation, 9,328 kB of additional disk space will be used.
Do you want to continue [Y/n]? y
[**DELETED LINES**]
Setting up libpq-dev (9.1.15-0ubuntu0.12.04) ...
Setting up libssl-doc (1.0.1-4ubuntu5.27) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
[root@whatever ~]#
[root@whatever ~]#
[root@whatever ~]# # OK Looks like libpq-dev is installed.
[root@whatever ~]# # Lets see if we hav been successful
[root@whatever ~]# #    Go back to joed.
[root@whatever ~]#
[root@whatever ~]# su joed
[joed@whatever ~]$ cd [path to SQL dir ie whatever dir has your C code in it]
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$ # Via WinSCP, I changed all permissions recursively to 755 Group: joed [1000] Owner: joed [1000] in [path to]/trial_test_matrix
[joed@whatever SQL]$ # see: http://www.postgresql.org/docs/9.1/interactive/auth-methods.html#AUTH-TRUST
[joed@whatever SQL]$ # see  http://www.postgresql.org/docs/9.3/static/libpq-connect.html
[joed@whatever SQL]$ #     added joed's pw for the omnia db to the connect string in tse_analogous_db.c
[joed@whatever SQL]$
[joed@whatever SQL]$ # OK lets Compile, Link & run tse_analogous_db
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$ # Compile tse_analogous_db.c
[joed@whatever SQL]$ gcc -c  -I/usr/include/postgresql  tse_analogous_db.c
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$ # Link tse_analogous_db.o into tse_analogous_db.exe
[joed@whatever SQL]$ gcc -o tse_analogous_db.exe tse_analogous_db.o -L/usr/lib -lpq
[joed@whatever SQL]$
[joed@whatever SQL]$
[joed@whatever SQL]$ # Run tse_analogous_db.exe
[joed@whatever SQL]$ ./tse_analogous_db.exe
datname        datdba         encoding       datcollate     datctype       datistemplate  datallowconn   datconnlimit   datlastsysoid  datfrozenxid   dattablespace  datacl

template1      10             6              en_US.UTF-8    en_US.UTF-8    t              t              -1             11942          706            1663           {=c/postgres,postgres=CTc/postgres}
template0      10             6              en_US.UTF-8    en_US.UTF-8    t              f              -1             11942          706            1663           {=c/postgres,postgres=CTc/postgres}
postgres       10             6              en_US.UTF-8    en_US.UTF-8    f              t              -1             11942          706            1663
omnia          10             6              en_US.UTF-8    en_US.UTF-8    f              t              -1             11942          706            1663


joe_t1 begin
id             test_ix        origin         max_prime      primes_tail_0  primes_tail_1  primes_tail_2  primes_tail_3  primes_tail_4  primes_tail_5  primes_tail_6  primes_tail_7

17             16             Aberdeen                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        830            0              0              0              0              0              0              0              0
func: func_uses_DAL_sub0, test_index: 0


func_uses_DAL_sub0.PQntuples(res): 1


[joed@whatever SQL]$
[joed@whatever SQL]$ # yahhoo SUCCESS
[joed@whatever SQL]$

Thus Endeth the Lesson :)

Love and peace,

Joe

Analyze Python cProfile stats created with pstats.dump_stats() off line.

Background

In order to improve Code Performance, I need to find functions that are good candidates for Cython implementation.

Here’s the link to the Python Docs for Code Profiling. Pay attention to cProfile.

I asked and answered this on stack overflow here.

Preamble

Previously I have essentially done the following:

import cProfile, pstats, StringIO
pr = cProfile.Profile()
pr.enable()
# ... my code did something ...
pr.disable()
s = StringIO.StringIO()
sortby = 'cumulative'
ps = pstats.Stats(pr, stream=s).sort_stats(sortby)

ps.dump_stats('stats.dmp')  # dump the stats to a file named stats.dmp

So now I have the file named ‘stats.dmp’ stored off line.

Here’s how I used pstats to analyze the ‘stats.dmp’ file for human consumption off line.

The Code

#!/usr/local/bin/python2.7
# -*- coding: UTF-8 -*-
"""analyze_dmp.py takes the file INFILEPATH [a pstats dump file] Producing OUTFILEPATH [a human readable python profile]
Usage:   analyze_dmp.py INFILEPATH  OUTFILEPATH
Example: analyze_dmp.py stats.dmp   stats.log
"""
import sys, os
import cProfile, pstats, StringIO

def analyze_dmp(myinfilepath='stats.dmp', myoutfilepath='stats.log'):
    out_stream = open(myoutfilepath, 'w')
    ps = pstats.Stats(myinfilepath, stream=out_stream)
    sortby = 'cumulative'

    ps.strip_dirs().sort_stats(sortby).print_stats(.3)  # plink around with this to get the results you need

NUM_ARGS = 2
def main():
    args = sys.argv[1:]
    if len(args) != NUM_ARGS or "-h" in args or "--help" in args:
        print __doc__
        s = raw_input('hit return to quit')
        sys.exit(2)
    analyze_dmp(myinfilepath=args[0], myoutfilepath=args[1])

if __name__ == '__main__':
    main()

I tested this with a .dmp file created on Linux & analyzed on Windows XP. It worked FINE. The above Python file is named, “analyze_dmp.py”.

An Additional Resource

I also found out how to use the Python command line to analyze stats.dmp in an interactive way here.

Here’s the command line i used on Windows XP.

ipython -m pstats stats.dmp

lxml.html basic parse methods for [url, file, string]

lxml.html examples of parsing from

  • URLs
  • Files
  • Strings

URLs

import lxml.html
htmltree = lxml.html.parse('http://joecodeswell.com')

htmltree.xpath("//title")[0].text

'''
OUTPUT:
'JoeCodeswell.com'
'''

Files

N.B. Save ‘http://joecodeswell.com&#8217; as a file named ‘JoeCodeswell.com.htm’.
Make sure to cd to the dir containing the file before running the following.

import lxml.html
htmltree = lxml.html.parse('JoeCodeswell.com.htm')

htmltree.xpath("//title")[0].text

'''
OUTPUT:
'JoeCodeswell.com'
'''

Strings

N.B. Save ‘http://joecodeswell.com&#8217; as a file named ‘JoeCodeswell.com.htm’.
Make sure to cd to the dir containing the file before running the following.

import lxml.html

f = open('JoeCodeswell.com.htm', 'r'); the_string = f.read(); f.close()
htmltree = lxml.html.fromstring(the_string)

htmltree.xpath("//title")[0].text

'''
OUTPUT:
'JoeCodeswell.com'
'''