Recent Updates

Joe 4:24 pm on October 2, 2018  

react post from wp-admin editor text tab


/**
 *  File: App.js
 *  "React State Example-1: Update state using 2 select elements."
 *  by Joe Dorocak (JoeCodeswell.com) 
 *  Demonstrates changing state with 2 select elements 
 * 
 *  Source Code Link: https://joecodeswell.wordpress.com/2018/09/25/react-state-example-1-update-state-using-2-select-elements/
 * 
 * MIT License
 * 
 * Copyright (c) [2018] [Joseph P. Dorocak]
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in all
 * copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 * 
 *  https://en.forums.wordpress.com/topic/posting-javascript-markdown-does-not-show/
 * 
 *  
 */

import React, { Component } from 'react';
import logo from './logo.svg';
import './App.css';

let programmerList =  ["Admiral Grace Hopper","Ada Lovelace","Donald Knuth","James Gosling","Guido van Rossum","Linus Torvalds"];

class App extends Component {
  constructor() {
    super();
    this.state = {
      counter: 0,
      programmer_list: programmerList,
      fmSelFrom: programmerList[0],
      fmSelTo:programmerList[0]
    };
  }
  handleFromChange = (event) => {
    this.setState({ 
      fmSelFrom: event.target.value     
    });
  }
  handleToChange = (event) => {
    this.setState({ 
      fmSelTo: event.target.value    
    });
  }
  render() {
    // inside class App -> render(){ before return(
    return (

      /* begin out of the box jsx */
      <div className="App">
        <header className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          <h1 className="App-title">Welcome to React</h1>
        </header>
        <p className="App-intro">
          To get started, edit <code>src/App.js</code> and save to reload.
        </p>
        {/* end out of the box jsx */}
        {/* begin non out of the box jsx */}
        <div id="show-programmer-list">
          <p>Programmer List</p>
            {this.state.programmer_list.map(  (itm,ix) => <p key={ix} value={itm} >ix:{ix}->itm:{itm}</p> )   }          
        </div>

        <div id="select-programmer-from-to">
        <label>
            Pick your FROM Programmer:        
            <select id="select-from"  value={this.value} onChange={this.handleFromChange} >
              {this.state.programmer_list.map(  (itm,ix) => <option key={ix} value={ix+":"+itm} >{itm}</option> )   }  
            </select>
        </label>
        <label>
          Pick your TO Programmer:         
          <select id="select-to" value={this.key} onChange={this.handleToChange} >
            {this.state.programmer_list.map(  (itm,ix) => <option key={ix} value={itm} >{itm}</option> )   }
          </select>
        </label>

        <p>this.state.fmSelFrom: {this.state.fmSelFrom}</p>
        <p>this.state.fmSelTo: {this.state.fmSelTo}</p>

        </div>
        {/* end non out of the box jsx */}
     
      </div> 
      /* end of <div className="App"> */

      // inside class App -> render(){ ->return( -> after <div className="App"></div>

    ); // end render -> return
  } // end render () code
}

export default App;
App.js.txt
Displaying App.js.txt.
Joe 4:08 pm on October 2, 2018  

Help with publishing code on WordPress.com

Here is a link to a WordPress.com support request of mine regarding publishing code on WordPress.com, posting “` javascript markdown DOES NOT show ”.

The folks were a GREAT HELP.

Thanks again to  @melancholia.

Love and peace,

Joe

Joe 8:15 pm on September 25, 2018  

React State Example-1: Update state using 2 select elements.

Based on Out of the box src for create-react-app -V 1.5.2


/**
 *  File: App.js
 *  "React State Example-1: Update state using 2 select elements."
 *  by Joe Dorocak (JoeCodeswell.com) 
 *  Demonstrates changing state with 2 select elements 
 * 
 *  Source Code Link: https://joecodeswell.wordpress.com/2018/09/25/react-state-example-1-update-state-using-2-select-elements/
 * 
 * MIT License
 * 
 * Copyright (c) [2018] [Joseph P. Dorocak]
 * 
 * Permission is hereby granted, free of charge, to any person obtaining a copy
 * of this software and associated documentation files (the "Software"), to deal
 * in the Software without restriction, including without limitation the rights
 * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 * copies of the Software, and to permit persons to whom the Software is
 * furnished to do so, subject to the following conditions:
 * 
 * The above copyright notice and this permission notice shall be included in all
 * copies or substantial portions of the Software.
 * 
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 * SOFTWARE.
 * 
 *  https://en.forums.wordpress.com/topic/posting-javascript-markdown-does-not-show/
 * 
 *  
 */

import React, { Component } from 'react';
import logo from './logo.svg';
import './App.css';

let programmerList =  ["Admiral Grace Hopper","Ada Lovelace","Donald Knuth","James Gosling","Guido van Rossum","Linus Torvalds"];

class App extends Component {
  constructor() {
    super();
    this.state = {
      counter: 0,
      programmer_list: programmerList,
      fmSelFrom: programmerList[0],
      fmSelTo:programmerList[0]
    };
  }
  handleFromChange = (event) => {
    this.setState({ 
      fmSelFrom: event.target.value     
    });
  }
  handleToChange = (event) => {
    this.setState({ 
      fmSelTo: event.target.value    
    });
  }
  render() {
    // inside class App -> render(){ before return(
    return (

      /* begin out of the box jsx */
      <div className="App">
        <header className="App-header">
          <img src={logo} className="App-logo" alt="logo" />
          <h1 className="App-title">Welcome to React</h1>
        </header>
        <p className="App-intro">
          To get started, edit <code>src/App.js</code> and save to reload.
        </p>
        {/* end out of the box jsx */}
        {/* begin non out of the box jsx */}
        <div id="show-programmer-list">
          <p>Programmer List</p>
            {this.state.programmer_list.map(  (itm,ix) => <p key={ix} value={itm} >ix:{ix}->itm:{itm}</p> )   }          
        </div>

        <div id="select-programmer-from-to">
        <label>
            Pick your FROM Programmer:        
            <select id="select-from"  value={this.value} onChange={this.handleFromChange} >
              {this.state.programmer_list.map(  (itm,ix) => <option key={ix} value={ix+":"+itm} >{itm}</option> )   }  
            </select>
        </label>
        <label>
          Pick your TO Programmer:         
          <select id="select-to" value={this.key} onChange={this.handleToChange} >
            {this.state.programmer_list.map(  (itm,ix) => <option key={ix} value={itm} >{itm}</option> )   }
          </select>
        </label>

        <p>this.state.fmSelFrom: {this.state.fmSelFrom}</p>
        <p>this.state.fmSelTo: {this.state.fmSelTo}</p>

        </div>
        {/* end non out of the box jsx */}
     
      </div> 
      /* end of <div className="App"> */

      // inside class App -> render(){ ->return( -> after <div className="App"></div>

    ); // end render -> return
  } // end render () code
}

// App.js.txt
// Displaying App.js.txt.

export default App;
Joe 5:01 pm on September 22, 2018
Tags: 222   

Out of the box src for create-react-app -V 1.5.2

App.js out of the box

import React, { Component } from 'react';
import logo from './logo.svg';
import './App.css';

class App extends Component {
  render() {
    return (
      <div>
        <header>
          <img src="{logo}" alt="logo" />
          <h1>Welcome to React</h1>
        </header>

          To get started, edit <code>src/App.js</code> and save to reload.

      </div>
    );
  }
}

export default App;

App.css out of the box

.App {
  text-align: center;
}

.App-logo {
  animation: App-logo-spin infinite 20s linear;
  height: 80px;
}

.App-header {
  background-color: #222;
  height: 150px;
  padding: 20px;
  color: white;
}

.App-title {
  font-size: 1.5em;
}

.App-intro {
  font-size: large;
}

@keyframes App-logo-spin {
  from { transform: rotate(0deg); }
  to { transform: rotate(360deg); }
}

App.test.js out of the box

import React from 'react';
import ReactDOM from 'react-dom';
import App from './App';

it('renders without crashing', () => {
  const div = document.createElement('div');
  ReactDOM.render(<App />, div);
  ReactDOM.unmountComponentAtNode(div);
});

index.css out of the box

body {
  margin: 0;
  padding: 0;
  font-family: sans-serif;
}

index.js out of the box

import React from 'react';
import ReactDOM from 'react-dom';
import './index.css';
import App from './App';
import registerServiceWorker from './registerServiceWorker';

ReactDOM.render(<App />, document.getElementById('root'));
registerServiceWorker();

registerServiceWorker.js out of the box

<br />// In production, we register a service worker to serve assets from local cache.

// This lets the app load faster on subsequent visits in production, and gives
// it offline capabilities. However, it also means that developers (and users)
// will only see deployed updates on the "N+1" visit to a page, since previously
// cached resources are updated in the background.

// To learn more about the benefits of this model, read https://goo.gl/KwvDNy.
// This link also includes instructions on opting out of this behavior.

const isLocalhost = Boolean(
  window.location.hostname === 'localhost' ||
    // [::1] is the IPv6 localhost address.
    window.location.hostname === '[::1]' ||
    // 127.0.0.1/8 is considered localhost for IPv4.
    window.location.hostname.match(
      /^127(?:\.(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}$/
    )
);

export default function register() {
  if (process.env.NODE_ENV === 'production' && 'serviceWorker' in navigator) {
    // The URL constructor is available in all browsers that support SW.
    const publicUrl = new URL(process.env.PUBLIC_URL, window.location);
    if (publicUrl.origin !== window.location.origin) {
      // Our service worker won't work if PUBLIC_URL is on a different origin
      // from what our page is served on. This might happen if a CDN is used to
      // serve assets; see https://github.com/facebookincubator/create-react-app/issues/2374
      return;
    }

    window.addEventListener('load', () => {
      const swUrl = `${process.env.PUBLIC_URL}/service-worker.js`;

      if (isLocalhost) {
        // This is running on localhost. Lets check if a service worker still exists or not.
        checkValidServiceWorker(swUrl);

        // Add some additional logging to localhost, pointing developers to the
        // service worker/PWA documentation.
        navigator.serviceWorker.ready.then(() => {
          console.log(
            'This web app is being served cache-first by a service ' +
              'worker. To learn more, visit https://goo.gl/SC7cgQ'
          );
        });
      } else {
        // Is not local host. Just register service worker
        registerValidSW(swUrl);
      }
    });
  }
}

function registerValidSW(swUrl) {
  navigator.serviceWorker
    .register(swUrl)
    .then(registration => {
      registration.onupdatefound = () => {
        const installingWorker = registration.installing;
        installingWorker.onstatechange = () => {
          if (installingWorker.state === 'installed') {
            if (navigator.serviceWorker.controller) {
              // At this point, the old content will have been purged and
              // the fresh content will have been added to the cache.
              // It's the perfect time to display a "New content is
              // available; please refresh." message in your web app.
              console.log('New content is available; please refresh.');
            } else {
              // At this point, everything has been precached.
              // It's the perfect time to display a
              // "Content is cached for offline use." message.
              console.log('Content is cached for offline use.');
            }
          }
        };
      };
    })
    .catch(error => {
      console.error('Error during service worker registration:', error);
    });
}

function checkValidServiceWorker(swUrl) {
  // Check if the service worker can be found. If it can't reload the page.
  fetch(swUrl)
    .then(response => {
      // Ensure service worker exists, and that we really are getting a JS file.
      if (
        response.status === 404 ||
        response.headers.get('content-type').indexOf('javascript') === -1
      ) {
        // No service worker found. Probably a different app. Reload the page.
        navigator.serviceWorker.ready.then(registration => {
          registration.unregister().then(() => {
            window.location.reload();
          });
        });
      } else {
        // Service worker found. Proceed as normal.
        registerValidSW(swUrl);
      }
    })
    .catch(() => {
      console.log(
        'No internet connection found. App is running in offline mode.'
      );
    });
}

export function unregister() {
  if ('serviceWorker' in navigator) {
    navigator.serviceWorker.ready.then(registration => {
      registration.unregister();
    });
  }
}

logo.svg out of the box

NOT shown due to WordPress.com markdown limitations!

#222

Joe 7:35 pm on January 16, 2018  

gDocInsertDateTime Google Docs add-on

For my own use & that of others, I just wrote gDocInsertDateTime. It is a Google Docs add-on that inserts a date-time stamp at the position of the cursor.

Please use the comments below for feedback. Thanks.

Shortlink: https://wp.me/p1zI3Z-uZ

Joe 3:36 pm on December 12, 2017  

I just installed P2 and…

FINALLY, I AM HAPPY! 🙂

My Travails

BUMMER: I just installed P2 and the links to my pages (menu?) at the top of the site disappeared! I am going to go back to Truly Minimal theme and see if they HOPEFULLY re appear!

ANOTHER BUMMER: wordpress says you can’t revert to an old theme.

ANOTHER BIG BIG BUMMER: Markdown gets replaced by [X]HTML in posts. Find Markdown onP2 by Automattic

ANOTHER BUMMER: STILL can’t get P2 to display links to my pages.
 See P2 User Guide

Fixing Menu to Pages

  1. My sites > JoeCodeswell > Themes [Customize button]
  2. Menus Tab
  3. Create New Menu [Named “Pages” N.B. name does not show]
  4. Menu Location: Where do you want this menu to appear? (If you plan to use a menu widget (opens in a new window), skip this step.) [X] Primary Menu (Current: Pages) N.B. I clicked [X] on this item.
  5. Click “+ Add Items” button RESULT Menu of links appears on right. Including as JSON: [Search menu items…, Pages: [Home,ES6 Notes, Django Notes, Python Notes, Web2py Notes, My Resume (SCROLL DOWN TO SEE!)], Posts, Categories, Tags, Format ]
  6. I clicked + on all the Page Items to add them to the menu

RESULT: Pages Menu APPEARS ACROSS TOP WITHOUT “Pages” title. Thanks be to God. Amen.

Joe 6:49 pm on December 3, 2016
Tags: code, csv, database, , sqlite   

Python code creates a database from .csv

Content

  1. Documentation
  2. Code

1. Documentation

csv2sqliteDB.py:
1. Creates an sqlite Database (from SQLITE_FILEPATH) then
2. Creates and Loads TABLE_NAME (adding id INTEGER PRIMARY KEY )
– FROM CSV_FILEPATH (with header, NO id field necessary)
– INTO DATABASE_FILEPATH
– CREATING DATABASE_FILEPATH if it doesn’t exist.

Usage:   csv2sqliteDB.py TABLE_NAME CSV_FILEPATH  SQLITE_FILEPATH
Example: csv2sqliteDB.py contacts   contacts2.csv contactsDB2id.db

2. Code


#!/usr/bin/python
#! python3
#! python2
# -*- coding: utf-8 -*-
"""csv2sqliteDB.py Creates and Loads TABLE_NAME (adding id INTEGER PRIMARY KEY )
      from CSV_FILEPATH (with header, NO id field necessary)
      into DATABASE_FILEPATH
          creating DATABASE_FILEPATH if it doesn't exist

Usage:   csv2sqliteDB.py TABLE_NAME CSV_FILEPATH  SQLITE_FILEPATH
Example: csv2sqliteDB.py contacts   contacts2.csv contactsDB2id.db
"""

# see:
    # Importing a CSV file into a sqlite3 - http://stackoverflow.com/a/2888042/601770
    # csv file using python with headers intact - http://stackoverflow.com/a/3428633/601770

import os, sys, csv, sqlite3

def csv2sqliteDB(table_name, csv_filepath, database_filepath):
    conn, cursor = createDatabaseIfNotExist(database_filepath)
    nlines = sum(1 for line in open(csv_filepath)) # http://stackoverflow.com/a/36973958/601770
    with open(csv_filepath, 'r') as f:
        reader = csv.reader(f)
        headerL = reader.__next__()

        createTable(table_name, conn, cursor, headerL)

        loadTable(table_name, conn, cursor, headerL, reader, nlines)
    conn.close()

def createDatabaseIfNotExist(database_filepath):
    # if we error, we rollback automatically, else commit!
    with sqlite3.connect(database_filepath) as conn:
        cursor = conn.cursor()
        cursor.execute('SELECT SQLITE_VERSION()')
        data = cursor.fetchone()
        print('SQLite version:', data)
        return conn, cursor

def createTable(table_name, conn, cursor,  headerL):
    """
    :param table_name:
    :param conn:
    :param cursor:
    :param header:
    :return:
    """
    # http://stackoverflow.com/a/12432311/601770
    # http: // stackoverflow.com / a / 19730169 / 601770

    textL = len(headerL)*'TEXT'.split()
    def_str = ', '.join( ['id INTEGER PRIMARY KEY'] + [i[0]+' '+i[1] for i in list(zip(headerL, textL))] )
    script_str = """DROP TABLE IF EXISTS %s;
        CREATE TABLE %s (%s);
    """%(table_name, table_name, def_str) # checks to see if table exists and makes a fresh table.

    cursor.executescript(script_str)


def loadTable(table_name, conn, cursor, headerL, reader, nlines):
    insert_str_model = "INSERT INTO {table_name} ({col_names}) VALUES({q_marks});"
    col_names = ','.join(headerL)
    q_marks = ','.join(len(headerL)*'?'.split())

    insert_str = insert_str_model.format(table_name=table_name, col_names=col_names, q_marks=q_marks)

    # DO THE INSERTS
    line_count = 0
    for row in reader:
        line_count += 1
        if line_count%10 == 1: print('line %s of %s'%(line_count, nlines))
        cursor.execute(insert_str, row)
        conn.commit()


NUM_ARGS = 3
def main():
    args = sys.argv[1:]
    if len(args) != NUM_ARGS or "-h" in args or "--help" in args:
        print (__doc__)
        sys.exit(2)
    csv2sqliteDB(args[0], args[1], args[2])

if __name__ == '__main__':
    main()

#code, #csv, #database, #python, #sqlite

Joe 7:49 pm on November 29, 2016
Tags: Coding, Debug Statement Execution, Education, , Mac, , Python IDLE, Raspberry Pi, Turtle Graphics,   

Teach Kids Coding – Turtle, Python, IDLE

Advantage – Immediate Feedback

As the student repeatedly presses the Debug Control’s “Over” button, she will be able to see the Turtle execute the commands she has given it.

Content

  • Summary
  • Video Link
  • Technique Steps
  • Code

Summary

This post will demonstrate a technique for teaching kids (or anyone) coding, using Turtle Graphics, Python, Python IDLE, Python IDLE’s Debug Control and Statement Execution Stepping.

This technique should work for Windows, Mac, and Linux (e.g. Raspberry Pi).

Video Link

Here is a link to a YouTube Video for this post.

Technique Steps

  1. Open your File Manager
    • “File Explorer” in Windows
    • “Finder” in Mac
    • “File Manager” in Raspberry Pi
  2. Navigate to the folder [CodeFolderPath] in which you have your code to be used. Most File Managers let you copy the path to a command line.
  3. Open a Command Line Window
    • “Command Prompt” in Windows
    • “Terminal” in Mac and Raspberry Pi
  4. At the Command Line, change directory to [CodeFolderPath], whatever that is.
    • example:
    • $ cd ~/pythonPjs/MasonPjs
  5. At the Command Line, run IDLE.
    • $ idle
    • This will bring up the IDLE Shell Window.
  6. In the IDLE Shell Window’s menu bar, click File Open and open the appropriate code file.
    • This will bring up the IDLE Code Window.
  7. In the IDLE Shell Window’s menu bar, click Debug/Debugger.
    • This will bring up the IDLE Debug Control Window.
  8. In the Debug Control Window:
    • UnTick the Stack Checkbox.
    • Tick the Source Checkbox.
  9. Minimize the IDLE Shell, File Manager and Command Line Windows. We want the student to focus on the important windows.
    • IDLE Debug Control
    • IDLE Code
    • Turtle Graphics
  10. Position the Debug Control & IDLE Code windows so you’ll be able to see them AND the Turtle Graphics Window SIMULTANEOUSLY. I plan on Turtle Graphics Window to be on the RIGHT of the screen. So I put:
    • Debug Contol – Top Left of the screen.
    • IDLE Code – Bottom Left of the screen. I make it about the same WIDTH as the Debug Control Window.
  11. In the IDLE Code window click on File/Run/Run Module
    • RESIZE it to be above the IDLE Code Window.
    • ALSO Minimize the IDLE Shell Window that (annoyingly 🙂 ) popped up again.
  12. Click on Debug Control Window’s “Over” button until the Turtle Graphics window appears. NOTICE with each click of “Over”.
    • NOTICE with each click, A NEW LINE in the Code Window is highlighted.
    • NOTICE when forward() or left() commands are executed, the Turtle Graphics Window responds with the appropriate action.
    • NOTICE after the line A1pos = pos(); A2abs_pos = abs(pos()) has been executed, the Debug Control Window contains 2 lines under the “Locals”
    • A1pos – (200.00,0.00) or something like this value
    • A2abs_pos – 200.00 or something like this value
      • Keeping track of the current values of these items.
    • NOTICE that after we execute the if abs(pos()) &lt; 1: statement, we loop back to just under the while True: statement.
  13. Click on "Over" for about 30 more times. NOTICE the Turtle Graphics window cooking along, OBEYING the commands in the Code Window. 🙂
  14. Right Click on the break Line & select "Set Breakpoint".
    • NOTICE the yellow background.
  15. In the Debug Control Window Click on the "Go" button. This will run until just before we execute the breakpoint line.
    • WATCH the Trutle Graphics window do its thing.
  16. Click "Over".
    • NOTICE we exit the while True: loop.
  17. Click "Over".
    • NOTICE the end_fill() statement colors the star yellow.
  18. Click "Over" one last time.
    • NOTICE we are DONE. 🙂

Code

Here is the code we used in the video.

from turtle import *
setup (width=500, height=500, startx=500)
color('red', 'yellow')
begin_fill()
while True:
    forward(200)
    left(170)
    A1pos = pos(); A2abs_pos = abs(pos()) 
    if abs(pos()) < 1:
        break
end_fill()
done()

#coding, #debug-statement-execution, #education, #linux, #mac, #python, #python-idle, #raspberry-pi, #turtle-graphics, #windows

Joe 4:33 pm on October 28, 2016  

Naive Spot-Check of AI Algorithms

More Adventures in AI

A “Back to Basics” RESET

I found the work of Jason Brownlee to help me get “Back to Basics” with AI.

Currently i am following his EXCELLENT Python Machine Learning Mini-Course.

At Lesson 9, He suggests one should “Spot-Check Algorithms”.

I took a naive approach employing the following “Analysis Code” section below to generate the resulting “Analysis Grid” section below.

I am hoping to use this post to get information in the Comments Section about how to use Data Preparation for various (Dataset, Model(Algorithm), Scoring) combinations, AND which (Dataset, Model(Algorithm), Scoring) combinations are JUST INCOMPATIBLE.

Methodology

To generate the Algorithm Spot-Check Cases i cross multiplied:

  1. (3) Datasets
    1.1 Boston House Price Data
    1.2 Iris Data
    1.3 Pima Indians Diabetes Data
  2. (4) Models(Algorithms)
    2.1 KNeighborsRegressor
    2.2 LinearRegression
    2.3 LogisticRegression
    2.4 LinearDiscriminantAnalysis
  3. (3) Scorings
    3.1 accuracy
    3.2 neg_mean_squared_error
    3.3 neg_log_loss

for a total of 36 Spot-Check Cases.

I then ran each case against a
kfold = sklearn.model_selection.KFold(n_splits=10, random_state=7)
with
results = sklearn.model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring)

Summary Analysis

There were 11 out of 36 Cases that returned numerical results. The other 25 Cases returned Errors or Warnings. The full Analysis can be seen in the “Analysis Grid” section below.

Question Reiteration, Answer, Further Study

Question Reiteration

How can I use Data Preparation for various (Dataset, Model(Algorithm), Scoring) combinations, AND which (Dataset, Model(Algorithm), Scoring) combinations are JUST INCOMPATIBLE.

I posted this question to Jason Brownlee in the Comments on Python Machine Learning Mini-Course. The Answer is his response to me there.

Answer

From Jason Brownlee in the Comments on Python Machine Learning Mini-Course:

Nice post and great question Joe.

Spot checking is to discover which algorithms look good on one given dataset. Not across datasets.

You may need to group algorithms by their expectations then prepare data for each group.

Most machine learning algorithms expect data to have numeric input values and an integer encoded or one hot encoded output value for classification. This is a good normalized view of a dataset to construct.

Here’s a tutorial that shows how to spot check 7 machine learning algorithms on one problem in Python, Spot-Check Regression Machine Learning Algorithms in Python with scikit-learn.

Further Study

From Jason’s Answer, above, I need to study:

  • Algorithm Groups
  • Algorithm Group Expectations
  • Data Preparation

Analysis Code

#! python3
# -*- coding: utf-8 -*-
"""analyzeAlgorithmsMod.py module contains  AnalyzeAlgorithms class.
    It aids in the comparison of Machine Learning Algorithms(Models)
       for a particular dataset.
"""
import os, sys, shutil, time, datetime, urllib, tarfile, zipfile, csv, io, copy, itertools, six

import numpy as np
import pandas as pd
import matplotlib
import matplotlib.pyplot as plt
import scipy.stats
import sklearn
import sklearn.preprocessing
import sklearn.linear_model

import sklearn.cross_validation
# import sklearn.model_selection

import sklearn.ensemble
import sklearn.metrics
import sklearn.discriminant_analysis
import sklearn.preprocessing
import itertools
import textwrap
from pprint import pprint as pp

import warnings
warnings.filterwarnings("error")


def reportInModule(model, scoring, results):
    print("\n\n{model: '%s', scoring: '%s'}:\n    results summary: %.3f mean (%.3f) std" % (type(model).__name__, scoring, results.mean(), results.std(),))
    print("    sorted(results):")
    pp(sorted(results), indent=8)


class AnalyzeAlgorithms:
    """Aids in the comparison of Machine Learning Algorithms
       for a particular dataset.
           http://machinelearningmastery.com/python-machine-learning-mini-course/
    """

    @staticmethod
    def calcX_Y(dataFrame):
        array = dataFrame.values
        X = array[:, 0:dataFrame.shape[1]-1]
        Y = array[:, dataFrame.shape[1]-1]
        return X, Y

    def __init__(self, datasetInfoTupleList, modelList, scoringStrList, kfold):
        self.datasetInfoTupleList = datasetInfoTupleList # (datasetTitle, csvFilePath, delim_whitespace, columnNamesList)
        self.modelList = modelList
        self.scoringStrList = scoringStrList
        self.kfold = kfold

    def analyzeAlgorithms(self):

        for datasetInfoTuple in self.datasetInfoTupleList: # (datasetTitle, csvFilePath, delim_whitespace, columnNamesList)

            self.datasetTitle, self.csvFilePath, self.delim_whitespace, self.columnNamesList = datasetInfoTuple
            self.df = pd.read_csv(self.csvFilePath, delim_whitespace=self.delim_whitespace, names=self.columnNamesList)
            self.X, self.Y = AnalyzeAlgorithms.calcX_Y(self.df)

            for model in self.modelList:

                for scoring in self.scoringStrList:
                    print("trying|%s|%s|%s|"%(self.datasetTitle, type(model).__name__, scoring),end='')

                    results = self.genResults(model, scoring, self.X, self.Y, self.kfold)
                    self.report_short(datasetInfoTuple, model, scoring, results)

    def genResults(self, model, scoring, X, Y, kfold):
        try:
            results = sklearn.model_selection.cross_val_score(model, X, Y, cv=kfold, scoring=scoring)
        except:
            results = "Error: %s"%( sys.exc_info()[1] )
        return results

    def report_short(self, datasetInfoTuple, model, scoring, results):
        datasetTitle = datasetInfoTuple[0]
        if isinstance(results, six.string_types):
            results_short = results.splitlines()[0]
            print(results_short)
        else:
            results_short = results.mean()
            print(results_short)




    def report(self, model, scoring, results):
        if isinstance(results, six.string_types):    # Error
            print("\n\n{model: '%s', scoring: '%s'}:\n    results ERROR: %s" % (type(model).__name__, scoring, results) )
        else:
            print("\n\n{model: '%s', scoring: '%s'}:\n    results summary: %.3f mean (%.3f) std" % (type(model).__name__, scoring, results.mean(), results.std(),))
            print("    sorted(results):")
            pp(sorted(results), indent=8)


def test():
    datasetInfoTupleList = [
        ('Boston House Price Data', # datasetTitle
             r'C:\BLA\BLA\BLA\data\BostonHousing\housing.data.txt', # csvFilePath
             True, # delim_whitespace for csv boolean
             ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV'], # columnNames
         ),
        ('Iris Data',  # datasetTitle
             r'C:\BLA\BLA\BLA\data\iris\iris.data.txt', # csvFilePath
             False,  # delim_whitespace for csv boolean
             ['sepalLength', 'sepalWidth', 'petalLength', 'petalWidth', 'class'],  # columnNames
        ),
        ('Pima Indians Diabetes Data',  # datasetTitle
             r'C:\BLA\BLA\BLA\data\PimaIndiansDiabetes\pima-indians-diabetes.data.txt',  # csvFilePath
             False,  # delim_whitespace for csv boolean
             ['preg', 'plas', 'pres', 'skin', 'test', 'mass', 'pedi', 'age', 'class'],  # columnNames
         ),

    ]

    modelList = [ sklearn.neighbors.KNeighborsRegressor(),
                  sklearn.linear_model.LinearRegression(),
                  sklearn.linear_model.LogisticRegression(),
                  sklearn.discriminant_analysis.LinearDiscriminantAnalysis(),
                  ]

    scoringStrList = [
                       'accuracy',
                       'neg_mean_squared_error',
                       'neg_log_loss',

                    ]
    kfold = sklearn.model_selection.KFold(n_splits=10, random_state=7)

    analysis02 = AnalyzeAlgorithms(datasetInfoTupleList, modelList, scoringStrList, kfold)
    analysis02.analyzeAlgorithms()




def main():
    test()

if __name__ == '__main__':
    main()

output = """

C:\Python35\python.exe C:/BLA/BLA/MachineLearningMasteryPj/AnalyzeAlgorithmsPkg/analyzeAlgorithmsMod.py
C:\Python35\lib\site-packages\sklearn\cross_validation.py:44: DeprecationWarning: This module was deprecated in version 0.18 in favor of the model_selection module into which all the refactored classes and functions are moved. Also note that the interface of the new CV iterators are different from that of this module. This module will be removed in 0.20.
  "This module will be removed in 0.20.", DeprecationWarning)
trying|Boston House Price Data|KNeighborsRegressor|accuracy|Error: continuous is not supported
trying|Boston House Price Data|KNeighborsRegressor|neg_mean_squared_error|-107.28683898
trying|Boston House Price Data|KNeighborsRegressor|neg_log_loss|Error: 'KNeighborsRegressor' object has no attribute 'predict_proba'
trying|Boston House Price Data|LinearRegression|accuracy|Error: continuous is not supported
trying|Boston House Price Data|LinearRegression|neg_mean_squared_error|-34.7052559445
trying|Boston House Price Data|LinearRegression|neg_log_loss|Error: 'LinearRegression' object has no attribute 'predict_proba'
trying|Boston House Price Data|LogisticRegression|accuracy|Error: Unknown label type: 'continuous'
trying|Boston House Price Data|LogisticRegression|neg_mean_squared_error|Error: Unknown label type: 'continuous'
trying|Boston House Price Data|LogisticRegression|neg_log_loss|Error: Unknown label type: 'continuous'
trying|Boston House Price Data|LinearDiscriminantAnalysis|accuracy|Error: Unknown label type: (array([ 20.5,  25. ,  23.4,  18.9,  35.4,  24.7,  31.6,  23.3,  19.6,
trying|Boston House Price Data|LinearDiscriminantAnalysis|neg_mean_squared_error|Error: Unknown label type: (array([ 20.5,  25. ,  23.4,  18.9,  35.4,  24.7,  31.6,  23.3,  19.6,
trying|Boston House Price Data|LinearDiscriminantAnalysis|neg_log_loss|Error: Unknown label type: (array([ 20.5,  25. ,  23.4,  18.9,  35.4,  24.7,  31.6,  23.3,  19.6,
trying|Iris Data|KNeighborsRegressor|accuracy|Error: unsupported operand type(s) for /: 'str' and 'int'
trying|Iris Data|KNeighborsRegressor|neg_mean_squared_error|Error: unsupported operand type(s) for /: 'str' and 'int'
trying|Iris Data|KNeighborsRegressor|neg_log_loss|Error: 'KNeighborsRegressor' object has no attribute 'predict_proba'
trying|Iris Data|LinearRegression|accuracy|Error: could not convert string to float: 'Iris-virginica'
trying|Iris Data|LinearRegression|neg_mean_squared_error|Error: could not convert string to float: 'Iris-virginica'
trying|Iris Data|LinearRegression|neg_log_loss|Error: could not convert string to float: 'Iris-virginica'
trying|Iris Data|LogisticRegression|accuracy|0.88
trying|Iris Data|LogisticRegression|neg_mean_squared_error|Error: could not convert string to float: 'Iris-setosa'
trying|Iris Data|LogisticRegression|neg_log_loss|Error: y_true contains only one label (Iris-setosa). Please provide the true labels explicitly through the labels argument.
trying|Iris Data|LinearDiscriminantAnalysis|accuracy|Error: The priors do not sum to 1. Renormalizing
trying|Iris Data|LinearDiscriminantAnalysis|neg_mean_squared_error|Error: The priors do not sum to 1. Renormalizing
trying|Iris Data|LinearDiscriminantAnalysis|neg_log_loss|Error: The priors do not sum to 1. Renormalizing
trying|Pima Indians Diabetes Data|KNeighborsRegressor|accuracy|Error: Can't handle mix of binary and continuous
trying|Pima Indians Diabetes Data|KNeighborsRegressor|neg_mean_squared_error|-0.196342447027
trying|Pima Indians Diabetes Data|KNeighborsRegressor|neg_log_loss|Error: 'KNeighborsRegressor' object has no attribute 'predict_proba'
trying|Pima Indians Diabetes Data|LinearRegression|accuracy|Error: Can't handle mix of binary and continuous
trying|Pima Indians Diabetes Data|LinearRegression|neg_mean_squared_error|-0.162812506544
trying|Pima Indians Diabetes Data|LinearRegression|neg_log_loss|Error: 'LinearRegression' object has no attribute 'predict_proba'
trying|Pima Indians Diabetes Data|LogisticRegression|accuracy|0.76951469583
trying|Pima Indians Diabetes Data|LogisticRegression|neg_mean_squared_error|-0.23048530417
trying|Pima Indians Diabetes Data|LogisticRegression|neg_log_loss|-0.492545522852
trying|Pima Indians Diabetes Data|LinearDiscriminantAnalysis|accuracy|0.773462064252
trying|Pima Indians Diabetes Data|LinearDiscriminantAnalysis|neg_mean_squared_error|-0.226537935748
trying|Pima Indians Diabetes Data|LinearDiscriminantAnalysis|neg_log_loss|-0.485655330102

Process finished with exit code 0


"""

Analysis Grid

Here’s the Analysis Grid copied from an Excel Spreadsheet.


| Dataset                    | Model                      | Scoring                | Result(Error or Mean(result)                                                                                                    | Joe's Comment                                                                     |
|----------------------------|----------------------------|------------------------|---------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------|
| Boston House Price Data    | KNeighborsRegressor        | accuracy               | Error: continuous is not supported                                                                                              | Data Prep? (Boston, [KNeighborsRegressor, LinearRegression], accuracy)            |
| Boston House Price Data    | KNeighborsRegressor        | neg_mean_squared_error | -107.286839                                                                                                                     |                                                                                   |
| Boston House Price Data    | KNeighborsRegressor        | neg_log_loss           | Error: 'KNeighborsRegressor' object has no attribute 'predict_proba'                                                            | NG: (KNeighborsRegressor, neg_log_loss)                                           |
| Boston House Price Data    | LinearRegression           | accuracy               | Error: continuous is not supported                                                                                              | Data Prep? (Boston, [KNeighborsRegressor, LinearRegression], accuracy)            |
| Boston House Price Data    | LinearRegression           | neg_mean_squared_error | -34.70525594                                                                                                                    |                                                                                   |
| Boston House Price Data    | LinearRegression           | neg_log_loss           | Error: 'LinearRegression' object has no attribute 'predict_proba'                                                               | Data Prep? Or Model-Scoring?([Boston, Pima], LinearRegression,   neg_log_loss)    |
| Boston House Price Data    | LogisticRegression         | accuracy               | Error: Unknown label type: 'continuous'                                                                                         | Data Prep (Boston, LogisticRegression, *)                                         |
| Boston House Price Data    | LogisticRegression         | neg_mean_squared_error | Error: Unknown label type: 'continuous'                                                                                         | Data Prep (Boston, LogisticRegression, *)                                         |
| Boston House Price Data    | LogisticRegression         | neg_log_loss           | Error: Unknown label type: 'continuous'                                                                                         | Data Prep (Boston, LogisticRegression, *)                                         |
| Boston House Price Data    | LinearDiscriminantAnalysis | accuracy               | Error: Unknown label type: (array([ 20.5,    25. ,  23.4,  18.9,    35.4,  24.7,  31.6,    23.3,  19.6,                         | Data Prep (Boston, LinearDiscriminantAnalysis *)                                  |
| Boston House Price Data    | LinearDiscriminantAnalysis | neg_mean_squared_error | Error: Unknown label type: (array([ 20.5,    25. ,  23.4,  18.9,    35.4,  24.7,  31.6,    23.3,  19.6,                         | Data Prep (Boston, LinearDiscriminantAnalysis *)                                  |
| Boston House Price Data    | LinearDiscriminantAnalysis | neg_log_loss           | Error: Unknown label type: (array([ 20.5,    25. ,  23.4,  18.9,    35.4,  24.7,  31.6,    23.3,  19.6,                         | Data Prep (Boston, LinearDiscriminantAnalysis *)                                  |
| Iris Data                  | KNeighborsRegressor        | accuracy               | Error: unsupported operand type(s) for /: 'str' and 'int'                                                                       | Data Prep: (Iris Data, KNeighborsRegressor,  [accuracy, neg_mean_squared_error] ) |
| Iris Data                  | KNeighborsRegressor        | neg_mean_squared_error | Error: unsupported operand type(s) for /: 'str' and 'int'                                                                       | Data Prep: (Iris Data, KNeighborsRegressor,  [accuracy, neg_mean_squared_error] ) |
| Iris Data                  | KNeighborsRegressor        | neg_log_loss           | Error: 'KNeighborsRegressor' object has no attribute 'predict_proba'                                                            | NG: (KNeighborsRegressor, neg_log_loss)                                           |
| Iris Data                  | LinearRegression           | accuracy               | Error: could not convert string to float: 'Iris-virginica'                                                                      | Data Prep: (Iris Data, LinearRegression, *)                                       |
| Iris Data                  | LinearRegression           | neg_mean_squared_error | Error: could not convert string to float: 'Iris-virginica'                                                                      | Data Prep: (Iris Data, LinearRegression, *)                                       |
| Iris Data                  | LinearRegression           | neg_log_loss           | Error: could not convert string to float: 'Iris-virginica'                                                                      | Data Prep: (Iris Data, LinearRegression, *)                                       |
| Iris Data                  | LogisticRegression         | accuracy               | 0.88                                                                                                                            |                                                                                   |
| Iris Data                  | LogisticRegression         | neg_mean_squared_error | Error: could not convert string to float: 'Iris-setosa'                                                                         |                                                                                   |
| Iris Data                  | LogisticRegression         | neg_log_loss           | Error: y_true contains only one   label (Iris-setosa). Please provide the true labels explicitly through the   labels argument. |                                                                                   |
| Iris Data                  | LinearDiscriminantAnalysis | accuracy               | Error: The priors do not sum to 1. Renormalizing                                                                                |                                                                                   |
| Iris Data                  | LinearDiscriminantAnalysis | neg_mean_squared_error | Error: The priors do not sum to 1. Renormalizing                                                                                |                                                                                   |
| Iris Data                  | LinearDiscriminantAnalysis | neg_log_loss           | Error: The priors do not sum to 1. Renormalizing                                                                                |                                                                                   |
| Pima Indians Diabetes Data | KNeighborsRegressor        | accuracy               | Error: Can't handle mix of binary and continuous                                                                                | Data Prep? (Pima, [KNeighborsRegressor, LinearRegression], accuracy)              |
| Pima Indians Diabetes Data | KNeighborsRegressor        | neg_mean_squared_error | -0.196342447                                                                                                                    |                                                                                   |
| Pima Indians Diabetes Data | KNeighborsRegressor        | neg_log_loss           | Error: 'KNeighborsRegressor' object has no attribute 'predict_proba'                                                            | NG: (KNeighborsRegressor, neg_log_loss)                                           |
| Pima Indians Diabetes Data | LinearRegression           | accuracy               | Error: Can't handle mix of binary and continuous                                                                                | Data Prep? (Pima, [KNeighborsRegressor, LinearRegression], accuracy)              |
| Pima Indians Diabetes Data | LinearRegression           | neg_mean_squared_error | -0.162812507                                                                                                                    |                                                                                   |
| Pima Indians Diabetes Data | LinearRegression           | neg_log_loss           | Error: 'LinearRegression' object has no attribute 'predict_proba'                                                               | Data Prep? Or Model-Scoring?([Boston, Pima], LinearRegression,   neg_log_loss)    |
| Pima Indians Diabetes Data | LogisticRegression         | accuracy               | 0.769514696                                                                                                                     |                                                                                   |
| Pima Indians Diabetes Data | LogisticRegression         | neg_mean_squared_error | -0.230485304                                                                                                                    |                                                                                   |
| Pima Indians Diabetes Data | LogisticRegression         | neg_log_loss           | -0.492545523                                                                                                                    |                                                                                   |
| Pima Indians Diabetes Data | LinearDiscriminantAnalysis | accuracy               | 0.773462064                                                                                                                     |                                                                                   |
| Pima Indians Diabetes Data | LinearDiscriminantAnalysis | neg_mean_squared_error | -0.226537936                                                                                                                    |                                                                                   |
| Pima Indians Diabetes Data | LinearDiscriminantAnalysis | neg_log_loss           | -0.48565533                                                                                                                     |                                                                                   |
Joe 5:33 pm on September 24, 2016
Tags: ai, Artificial Intelligence,   

H2O.ai Adventures in Artificial Intelligence (ai)

Background

Although I HAVE NOT thought about Artificial Intelligence, ai, since i was a student in Michael Arbib’s class studying for my M.S., when i became aware of H2O.ai.com, i decided it was time to jump in. 🙂

The following will be a chronicle of my adventures. 🙂

THIS IS A WORK IN PROGRESS

Big Data Hadoop vs Apache Spark

Downloads: (H2O vs Sparkling Water)

H2O.ai’s offerings, H2O and Sparkling Water, seemed to pose the question, “What Big Data platform should I choose, Hadoop or Apache Spark?” I have learned that they are not competitors. Katherine Noyes says in Infoworld,

“They do different things. … Hadoop is essentially a distributed data infrastructure … Spark, on the other hand, is a data-processing tool that operates on those distributed data collection”.

OK. But which of H2O.ai’s Downloads, only 2 when i started, should i choose to investigate? I picked Sparkling Water because of a page explaining the ai “Classification” Use Case.

Goal Install & RUN PySparkling

Here’s some notes for PySparkling installation on Windows 10.
Be prepared for (SysAdmin, SysAdmin, … more SysAAdmin)!

Install Apache Spark (to use PySpark)

  • Apache Spark needs to be installed first
    • Downloads\ApacheSpark\spark-1.6.2-bin-hadoop2.6
    • > echo %PYSPARK_PYTHON% == C:\Python27\python.exe
    • Test Run PySpark in the PySpark Shell
    • > cd Downloads\ApacheSpark\spark-1.6.2-bin-hadoop2.6
    • > .\bin\pyspark.cmd
    • Test with QuickStart N.B. Click the Python_Tab
    • RESULT: OK
    • Test Run PySpark as a Self-Contained Application
    • Test with Self-Contained Application N.B. Click the Python_Tab
    • RESULT: NO GOOD –

Self-Contained PySpark RESULT

Here’s the Self-Contained RESULT with NO MODIFICATIONS of the sys.path

"""SimpleApp.py"""
from pyspark import SparkContext

logFile = "YOUR_SPARK_HOME/README.md"  # Should be some file on your system
sc = SparkContext("local", "Simple App")
logData = sc.textFile(logFile).cache()

numAs = logData.filter(lambda s: 'a' in s).count()
numBs = logData.filter(lambda s: 'b' in s).count()

print("Lines with a: %i, lines with b: %i" % (numAs, numBs))
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-1-024ac6cc8be6> in <module>()
      1 """SimpleApp.py"""
----> 2 from pyspark import SparkContext
      3 
      4 logFile = "YOUR_SPARK_HOME/README.md"  # Should be some file on your system
      5 sc = SparkContext("local", "Simple App")

ImportError: No module named pyspark

sys.path Analysis – PySpark Shell vs Plain Python27

<br /># This is pyspark shell sys.path
pysparkShellSysPath = '''
C:\Users\joeco\AppData\Local\Temp\spark-a00a2ab4-63a0-404f-b607-5f34c4206e76\userFiles-5a9b86bf-a518-4fef-b4de-b42005b143d5
C:\Python27\lib\site-packages\pywebview-0.8.2-py2.7.egg
C:\Python27\lib\site-packages\ouimeaux-0.7.9.post0-py2.7.egg
C:\Python27\lib\site-packages\gevent_socketio-0.3.6-py2.7.egg
C:\Python27\lib\site-packages\flask_restful-0.3.5-py2.7.egg
C:\Python27\lib\site-packages\pysignals-0.1.2-py2.7.egg
C:\Python27\lib\site-packages\pyyaml-3.11-py2.7-win-amd64.egg
C:\Python27\lib\site-packages\requests-2.9.1-py2.7.egg
C:\Python27\lib\site-packages\gevent-1.1rc3-py2.7-win-amd64.egg
C:\Python27\lib\site-packages\gevent_websocket-0.9.5-py2.7.egg
C:\Python27\lib\site-packages\pytz-2015.7-py2.7.egg
C:\Python27\lib\site-packages\six-1.10.0-py2.7.egg
C:\Python27\lib\site-packages\flask-0.10.1-py2.7.egg
C:\Python27\lib\site-packages\aniso8601-1.1.0-py2.7.egg
C:\Python27\lib\site-packages\greenlet-0.4.9-py2.7-win-amd64.egg
C:\Python27\lib\site-packages\itsdangerous-0.24-py2.7.egg
C:\Python27\lib\site-packages\werkzeug-0.11.3-py2.7.egg
C:\Python27\lib\site-packages\python_dateutil-2.4.2-py2.7.egg
C:\Python27\lib\site-packages\python_registry-1.1.0-py2.7.egg
C:\Python27\lib\site-packages\enum34-1.1.2-py2.7.egg
C:\Python27\lib\site-packages\speedtest_cli-0.3.4-py2.7.egg
C:\Python27\lib\site-packages\midi-0.2.3-py2.7.egg
C:\Python27\lib\site-packages\h2o_pysparkling_1.6-1.6.5-py2.7.egg
C:\Python27\lib\site-packages\tabulate-0.7.5-py2.7.egg
C:\Python27\lib\site-packages\future-0.15.2-py2.7.egg
C:\Users\joeco\Downloads\ApacheSpark\spark-1.6.2-bin-hadoop2.6\python\lib\py4j-0.9-src.zip
C:\Users\joeco\Downloads\ApacheSpark\spark-1.6.2-bin-hadoop2.6\python
C:\Users\joeco\Downloads\ApacheSpark\spark-1.6.2-bin-hadoop2.6
C:\WINDOWS\SYSTEM32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
C:\Python27\lib\site-packages\win32
C:\Python27\lib\site-packages\win32\lib
C:\Python27\lib\site-packages\Pythonwin
C:\Python27\lib\site-packages\wx-3.0-msw
'''


# This is plain python 2.7 sys.path
plainPythonSysPath = '''
C:\Python27\lib\site-packages\pywebview-0.8.2-py2.7.egg
C:\Python27\lib\site-packages\ouimeaux-0.7.9.post0-py2.7.egg
C:\Python27\lib\site-packages\gevent_socketio-0.3.6-py2.7.egg
C:\Python27\lib\site-packages\flask_restful-0.3.5-py2.7.egg
C:\Python27\lib\site-packages\pysignals-0.1.2-py2.7.egg
C:\Python27\lib\site-packages\pyyaml-3.11-py2.7-win-amd64.egg
C:\Python27\lib\site-packages\requests-2.9.1-py2.7.egg
C:\Python27\lib\site-packages\gevent-1.1rc3-py2.7-win-amd64.egg
C:\Python27\lib\site-packages\gevent_websocket-0.9.5-py2.7.egg
C:\Python27\lib\site-packages\pytz-2015.7-py2.7.egg
C:\Python27\lib\site-packages\six-1.10.0-py2.7.egg
C:\Python27\lib\site-packages\flask-0.10.1-py2.7.egg
C:\Python27\lib\site-packages\aniso8601-1.1.0-py2.7.egg
C:\Python27\lib\site-packages\greenlet-0.4.9-py2.7-win-amd64.egg
C:\Python27\lib\site-packages\itsdangerous-0.24-py2.7.egg
C:\Python27\lib\site-packages\werkzeug-0.11.3-py2.7.egg
C:\Python27\lib\site-packages\python_dateutil-2.4.2-py2.7.egg
C:\Python27\lib\site-packages\python_registry-1.1.0-py2.7.egg
C:\Python27\lib\site-packages\enum34-1.1.2-py2.7.egg
C:\Python27\lib\site-packages\speedtest_cli-0.3.4-py2.7.egg
C:\Python27\lib\site-packages\midi-0.2.3-py2.7.egg
C:\Python27\lib\site-packages\h2o_pysparkling_1.6-1.6.5-py2.7.egg
C:\Python27\lib\site-packages\tabulate-0.7.5-py2.7.egg
C:\Python27\lib\site-packages\future-0.15.2-py2.7.egg
C:\WINDOWS\SYSTEM32\python27.zip
C:\Python27\DLLs
C:\Python27\lib
C:\Python27\lib\plat-win
C:\Python27\lib\lib-tk
C:\Python27
C:\Python27\lib\site-packages
C:\Python27\lib\site-packages\win32
C:\Python27\lib\site-packages\win32\lib
C:\Python27\lib\site-packages\Pythonwin
C:\Python27\lib\site-packages\wx-3.0-msw
'''

# print('hello', len( sorted(pysprkShellSysPath.splitlines()) ) )


pysparksp = sorted(pysparkShellSysPath.splitlines())
len(pysparksp)
plainsp = sorted(plainPythonSysPath.splitlines())
len(plainsp)
for i in range( max(len(pysparksp),len(plainsp)) ):
    if i < len(pysparksp):  print ('pyspk', pysparksp[i])
    if i < len(plainsp  ):  print ('plain', plainsp[i])
    print 

----------------------OUTPUT---------------------------
('pyspk', '')
('plain', '')

('pyspk', 'C:\\Python27')
('plain', 'C:\\Python27')

('pyspk', 'C:\\Python27\\DLLs')
('plain', 'C:\\Python27\\DLLs')

('pyspk', 'C:\\Python27\\lib')
('plain', 'C:\\Python27\\lib')

('pyspk', 'C:\\Python27\\lib\\lib-tk')
('plain', 'C:\\Python27\\lib\\lib-tk')

('pyspk', 'C:\\Python27\\lib\\plat-win')
('plain', 'C:\\Python27\\lib\\plat-win')

('pyspk', 'C:\\Python27\\lib\\site-packages')
('plain', 'C:\\Python27\\lib\\site-packages')

('pyspk', 'C:\\Python27\\lib\\site-packages')
('plain', 'C:\\Python27\\lib\\site-packages')

('pyspk', 'C:\\Python27\\lib\\site-packages\x07niso8601-1.1.0-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\x07niso8601-1.1.0-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\tabulate-0.7.5-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\tabulate-0.7.5-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\x0clask-0.10.1-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\x0clask-0.10.1-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\x0clask_restful-0.3.5-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\x0clask_restful-0.3.5-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\x0cuture-0.15.2-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\x0cuture-0.15.2-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\Pythonwin')
('plain', 'C:\\Python27\\lib\\site-packages\\Pythonwin')

('pyspk', 'C:\\Python27\\lib\\site-packages\\enum34-1.1.2-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\enum34-1.1.2-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\gevent-1.1rc3-py2.7-win-amd64.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\gevent-1.1rc3-py2.7-win-amd64.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\gevent_socketio-0.3.6-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\gevent_socketio-0.3.6-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\gevent_websocket-0.9.5-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\gevent_websocket-0.9.5-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\greenlet-0.4.9-py2.7-win-amd64.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\greenlet-0.4.9-py2.7-win-amd64.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\h2o_pysparkling_1.6-1.6.5-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\h2o_pysparkling_1.6-1.6.5-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\itsdangerous-0.24-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\itsdangerous-0.24-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\midi-0.2.3-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\midi-0.2.3-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\ouimeaux-0.7.9.post0-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\ouimeaux-0.7.9.post0-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\pysignals-0.1.2-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\pysignals-0.1.2-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\python_dateutil-2.4.2-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\python_dateutil-2.4.2-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\python_registry-1.1.0-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\python_registry-1.1.0-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\pytz-2015.7-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\pytz-2015.7-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\pywebview-0.8.2-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\pywebview-0.8.2-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\pyyaml-3.11-py2.7-win-amd64.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\pyyaml-3.11-py2.7-win-amd64.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\six-1.10.0-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\six-1.10.0-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\speedtest_cli-0.3.4-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\speedtest_cli-0.3.4-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\werkzeug-0.11.3-py2.7.egg')
('plain', 'C:\\Python27\\lib\\site-packages\\werkzeug-0.11.3-py2.7.egg')

('pyspk', 'C:\\Python27\\lib\\site-packages\\win32')
('plain', 'C:\\Python27\\lib\\site-packages\\win32')

('pyspk', 'C:\\Python27\\lib\\site-packages\\win32\\lib')
('plain', 'C:\\Python27\\lib\\site-packages\\win32\\lib')

('pyspk', 'C:\\Python27\\lib\\site-packages\\wx-3.0-msw')
('plain', 'C:\\Python27\\lib\\site-packages\\wx-3.0-msw')

('pyspk', 'C:\\Users\\joeco\\AppData\\Local\\Temp\\spark-a00a2ab4-63a0-404f-b607-5f34c4206e76\\userFiles-5a9b86bf-a518-4fef-b4de-b42005b143d5')
('plain', 'C:\\WINDOWS\\SYSTEM32\\python27.zip')

('pyspk', 'C:\\Users\\joeco\\Downloads\\ApacheSpark\\spark-1.6.2-bin-hadoop2.6')
('plain', 'equests-2.9.1-py2.7.egg')

('pyspk', 'C:\\Users\\joeco\\Downloads\\ApacheSpark\\spark-1.6.2-bin-hadoop2.6\\python')

('pyspk', 'C:\\Users\\joeco\\Downloads\\ApacheSpark\\spark-1.6.2-bin-hadoop2.6\\python\\lib\\py4j-0.9-src.zip')

('pyspk', 'C:\\WINDOWS\\SYSTEM32\\python27.zip')

('pyspk', 'equests-2.9.1-py2.7.egg')

My Question: Am I missing some secret sauce to set the sys.path?

Answer: I don’t know yet.

Pursuing the Secret, sys.path, Sauce

This morning i went back to my Original Goal, run PySparkling, NOT PySpark, but PySparkling.

  • I found a new download page for Sparkling Water, PySparkling’s Uncle? 🙂
  • I chose my Spark version 1.6 out of [1.4, 1.5, 1.6]
  • Went to the 1.6 Download Page, downloaded Sparkling Water and clicked on the Python Tab which said “Get started with PySparkling”. Hallelulia! 🙂

Get started with PySparkling Steps

  1. Download Spark
    1.1 DONE
  2. Point SPARK_HOME to the existing installation of Spark and export variable MASTER. 
    >echo %SPARK_HOME%
...\Downloads\ApacheSpark\spark-1.6.2-bin-hadoop2.6

>echo %MASTER%
local-cluster[3,2,1024]

>
  3. From your terminal, run: 
    #To start an interactive Python terminal-
bin/pysparkling

PROBLEM: No bin/pysparkling.cmd

BUT sparkling-env.cmd exists

  • Could bin/pysparkling or bin/sparkling-env.cmd contain Secret sys.path Sauce?
  • I Need a break from Sys Admin
  • I Need TO CODE SOMETHING

MORE NEXT TIME! 🙂

TODO – FOLLOWING NEEDS WORK = UNPUBLISHED

PySparkling Installation

N.B. Click the Python_Tab

  • Be careful of your python environment.
  • I am running 2 & 3.
  • PySparkling needs 2.
C:\Users\joeco>python
Python 3.5.1 (v3.5.1:37a07cee5969, Dec  6 2015, 01:54:25) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysparkling import Context
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python35\lib\site-packages\h2o_pysparkling_1.6-1.6.5-py3.5.egg\pysparkling\__init__.py", line 11, in <module>
    from pysparkling.context import H2OContext
  File "C:\Python35\lib\site-packages\h2o_pysparkling_1.6-1.6.5-py3.5.egg\pysparkling\context.py", line 142
    print self
             ^
SyntaxError: Missing parentheses in call to 'print'
>>>
  • Be careful of your version of Sparkling Water
  • Be careful of your version of Spark

#ai, #artificial-intelligence, #python