Monthly Archives: August 2018

Data Scientist Tutorials

Videos

Data Science Hands-on with Open Source Tools (Archived)
Data Science Tools
PyCon Cleveland 2018 – Python For Data Science (2018 Intro Course)

Text-Tutorials

Google’s Python Class (Python 2.7)
Python Practice Book (Python 2.7)
Python Exercises, Practice, Solution (Python 3)
Practice Python – Beginner Python exercises (Python 3)
Interaktives Python 3 Tutorial mit über 100 Übungen
CodingBat code practice – Python
Python School

PyCharm Edu
Learn X in Y minutes, Where X=python3

Real Python

Python Learning Paths
Machine Learning With Python
An Intro to Threading in Python
Speed Up Your Python Program With Concurrency

Docker in Action – Fitter, Happier, More Productive
Object-Oriented Programming (OOP) in Python 3

Improve Skills by Practice and Problem Solving

wiki.python.org/moin/ProblemSets
exercism.io – Python Track (exercism.io CLI)
Rosalind Bioinformatics

$ exercism download --track=python --exercise=hamming

Downloaded to
/home/andreas/exercism/python/hamming

Jupyter and IPython

Installation

Package: jupyter-notebook (5.2.2-1)
Package: jupyter-notebook (5.4.1-1)

Documentation

Jupyter/IPython Notebook Quick Start Guide
Running the Jupyter Notebook
The Jupyter Notebook – Opening notebooks
How IPython works (deprecated)
Making kernels for IPython (deprecated)

JetBrains PyCharm

Using IPython/Jupyter Notebook with PyCharm

Create Virtualenv Environment

Using jupyter notebooks with a virtual environment
remove kernel on jupyter notebook

$ python -m venv projectname
$ source projectname/bin/activate
(venv) $ pip install ipykernel
(venv) $ ipython kernel install --user --name=projectname

MathJax

Swiss Python Summit 2018

Python Summit Talk Recordings

Peter Hoffmann – 12 Factor Apps for Data-Science with Python

Books

Python for Data Analysis, 2nd Edition
IPython Interactive Computing and Visualization Cookbook, Second Edition
Jupyter for Data Science

Software as a service (SaaS)

  • Cloud Computing
  • Infrastructure as a Service (IaaS)
  • Platform as a Service (PaaS)
  • Desktop as a Service (DaaS)
  • Managed software as a Service (MSaaS)
  • Mobile backend as a Service (MBaaS)
  • Information Technology Management as a Service (ITMaaS)

Centralized hosting of business applications dates back to the 1960s. Starting in that decade, IBM and other mainframe providers conducted a service bureau business, often referred to as time-sharing or utility computing. Such services included offering computing power and database storage to banks and other large organizations from their worldwide data centers.

Snowflake, The Only Data Warehouse Built for the Cloud
Welcome to the Snowflake Documentation
Key Concepts & Architecture

Snowflake is an analytic data warehouse provided as Software-as-a-Service (SaaS). Snowflake provides a data warehouse that is faster, easier to use, and far more flexible than traditional data warehouse offerings.

Snowflake’s data warehouse is not built on an existing database or “big data” software platform such as Hadoop. The Snowflake data warehouse uses a new SQL database engine with a unique architecture designed for the cloud. To the user, Snowflake has many similarities to other enterprise data warehouses, but also has additional functionality and unique capabilities.

The Jupyter Notebook, Formerly known as the IPython Notebook
The Jupyter Notebook
github.com/jupyter/jupyter/wiki/Jupyter-kernels, Jupyter kernels
How do I install different languages in Jupyter Notebook?

Data Scientist / Services for Machine Learning

Jupyter

web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text

pandas

Apache Parquet

Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language.

Apache Arrow

A cross-language development platform for in-memory data.
Reading and Writing the Apache Parquet Format

Amit Kumar – Let’s Talk About GIL!

SymPy

SymPy is a Python library for symbolic mathematics. It aims to become a full-featured computer algebra system (CAS) while keeping the code as simple as possible in order to be comprehensible and easily extensible. SymPy is written entirely in Python.

Global Interpreter Lock (GIL)

Java 9

Which Is the Hottest UI Framework in the Java World: JSF or JavaFX?

java.nio: Non-blocking I/O (Java)
Java NIO Tutorial
Java SE 8: Creating a Basic REST Web Service using Grizzly, Jersey, and Maven
Project Grizzly (software)
Project Grizzly, NIO Event Development Simplified

3D

LWJGL

LWJGL – Lightweight Java Game Library 3
github.com/LWJGL/lwjgl3, LWJGL is a Java library that enables cross-platform access to popular native APIs
github.com/LWJGLX/lwjgl3-awt, AWT support for LWJGL3

LWJGL3 based on GLFW
GLFW
Wikipedia: GLFW, small C library that allows the creation and management of windows with OpenGL contexts (like FreeGLUT or SDL)

How to use OpenGL in JavaFX?
JFXGL, Glue code that allows you to use JavaFX in your OpenGL/LWJGL3 app.

JOGL

Java OpenGL (JOGL)
JogAmp

JavaFX

Wikipedia: JavaFX
What Is JavaFX?
JDK 11 update: JavaFX will be decoupled from the JDK
What are the alternatives to JavaFX?
Gluon Scene Builder

One guys perspective on JavaFX, March 27, 2018
ControlsFX
Scenic View

Java 9: Ein erster Blick

Willkommen zu “Java 9: Ein erster Blick

Christopher Janietz

Einführung in Java 9: Geschichte, Community, JDK

  • Die Neuerungen von Java 9 im Überblick
  • Java-Community-Prozess
  • JDK-Installation
  • REPL mit Jshell

Modulsystem und Module in Java 9

  • Das Modulsystem
  • Hello World-Modul
  • Exports und Requires
  • Die Modulsyntax
  • Module im JDK
  • Reflection und Open
  • Service-Loader

Die HTTP/2-Schnittstelle in Java 9

  • HTTP/2 im Überblick
  • Request und Response
  • Websockets

Reactive Streams

  • Konzept von Reactive Streams
  • Reactive Streams im JDK
  • Processor

API-Erweiterungen und weitere Änderungen

  • Collection-API
  • Prozess-API
  • JVM-Logging

HP EliteBook 850 G4 Advanced – Linux Mint 19

General

sudo apt-get install mint-meta-codecs
sudo apt-get install nvme-cli
sudo apt-get install smartmontools
sudo apt-get install openssh-server
sudo apt-get install p7zip-full

Development

sudo apt-get install build-essential
sudo apt-get install coreutils
sudo apt-get install autoconf automake groff libtool

Python

$ sudo apt-get install python-gi python-gi-cairo python3-gi python3-gi-cairo gir1.2-gtk-3.0
$ sudo apt-get install libcairo2-dev
$ sudo apt-get install libgirepository1.0-dev

Qt

Ubuntu Users Wiki: Qt
Qt for Linux/X11
Install Qt 5 on Ubuntu (Ubuntu 12.10!!)
Getting Started on the Commandline

sudo apt-get install libfontconfig1
sudo apt-get install mesa-common-dev
sudo apt-get install libglu1-mesa-dev -y
sudo apt-get install libglvnd-dev
sudo apt-get install qtchooser

sudo apt-get install qt5-default
sudo apt-get install qttools5-dev-tools

$ apt-cache search 
libglvnd-dev - Vendor neutral GL dispatch library -- development files

$ qmake -project
$ qmake
Info: creating stash file /home/andreas/src/qt_libgl/.qmake.stash

$ make
g++ -c -pipe -O2 -Wall -W -D_REENTRANT -fPIC -DQT_DEPRECATED_WARNINGS -DQT_NO_DEBUG -DQT_GUI_LIB -DQT_CORE_LIB -I. -I. -isystem /usr/include/x86_64-linux-gnu/qt5 -isystem /usr/include/x86_64-linux-gnu/qt5/QtGui -isystem /usr/include/x86_64-linux-gnu/qt5/QtCore -I. -isystem /usr/include/libdrm -I/usr/lib/x86_64-linux-gnu/qt5/mkspecs/linux-g++ -o qt_libgl.o qt_libgl.cc
g++ -Wl,-O1 -o qt_libgl qt_libgl.o   -lQt5Gui -lQt5Core -lGL -lpthread 

Python

$ sudo apt-get install python-pip

$ sudo apt-get install python3-pip python3-dev python-virtualenv
$ sudo apt-get install python3-distutils

$ sudo apt-get install python3-setuptools python3-wheel
python3-setuptools python3-wheel

$ sudo apt-get install jupyter jupyter-notebook
  fonts-font-awesome javascript-common jupyter jupyter-client jupyter-console jupyter-core jupyter-nbformat jupyter-notebook \
  libjs-backbone libjs-bootstrap libjs-bootstrap-tour libjs-codemirror libjs-es6-promise libjs-jed libjs-jquery \
  libjs-jquery-typeahead libjs-jquery-ui libjs-marked libjs-moment libjs-requirejs libjs-requirejs-text libjs-text-encoding \
  libjs-underscore libjs-xterm python3-bleach python3-dateutil python3-decorator python3-entrypoints \
  python3-html5lib python3-ipykernel python3-ipython python3-ipython-genutils python3-jinja2 python3-jsonschema \
  python3-jupyter-client python3-jupyter-console python3-jupyter-core python3-mistune python3-nbconvert python3-nbformat \
  python3-notebook python3-pandocfilters python3-pickleshare python3-prompt-toolkit python3-pygments python3-simplegeneric \
  python3-terminado python3-testpath python3-tornado python3-traitlets python3-wcwidth python3-webencodings \
  python3-zmq

Java

$ sudo apt-get install openjdk-11-jdk

$ java --version
openjdk 10.0.2 2018-07-17
OpenJDK Runtime Environment (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1)
OpenJDK 64-Bit Server VM (build 10.0.2+13-Ubuntu-1ubuntu0.18.04.1, mixed mode)
# inxi -Fxz
System:    Host: andreas-HP-EliteBook-850-G4 Kernel: 4.15.0-32-generic x86_64 bits: 64 gcc: 7.3.0
           Desktop: Cinnamon 3.8.8 (Gtk 3.22.30-1ubuntu1) Distro: Linux Mint 19 Tara
Machine:   Device: laptop System: HP product: HP EliteBook 850 G4 serial: <filter>
           Mobo: HP model: 828C v: KBC Version 45.41 serial: <filter> UEFI: HP v: P78 Ver. 01.18 date: 05/16/2018
Battery    BAT0: charge: 35.7 Wh 98.3% condition: 36.4/36.4 Wh (100%) model: Hewlett-Packard Primary status: N/A
           hidpp__0: charge: 60% condition: NA/NA Wh model: Logitech M705 status: Discharging
CPU:       Dual core Intel Core i7-7500U (-MT-MCP-) arch: Kaby Lake rev.9 cache: 4096 KB
           flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx) bmips: 11616
           clock speeds: max: 3500 MHz 1: 798 MHz 2: 782 MHz 3: 799 MHz 4: 797 MHz
Graphics:  Card-1: Intel HD Graphics 620 bus-ID: 00:02.0
           Card-2: Advanced Micro Devices [AMD/ATI] Opal XT [Radeon R7 M265] bus-ID: 03:00.0
           Display Server: X.Org 1.19.6 drivers: modesetting,ati,radeon (unloaded: fbdev,vesa)
           Resolution: 3840x2160@60.00hz
           OpenGL: renderer: Mesa DRI Intel HD Graphics 620 (Kaby Lake GT2)
           version: 4.5 Mesa 18.0.5 Direct Render: Yes
Audio:     Card Intel Sunrise Point-LP HD Audio driver: snd_hda_intel bus-ID: 00:1f.3
           Sound: Advanced Linux Sound Architecture v: k4.15.0-32-generic
Network:   Card-1: Intel Ethernet Connection (4) I219-V driver: e1000e v: 3.2.6-k bus-ID: 00:1f.6
           IF: enp0s31f6 state: up speed: 100 Mbps duplex: full mac: <filter>
           Card-2: Intel Wireless 8265 / 8275 driver: iwlwifi bus-ID: 02:00.0
           IF: wlp2s0 state: down mac: <filter>
Drives:    HDD Total Size: 512.1GB (4.3% used)
           ID-1: /dev/nvme0n1 model: THNSN5512GPUK_TOSHIBA size: 512.1GB temp: 40C
Partition: ID-1: / size: 137G used: 6.4G (5%) fs: ext4 dev: /dev/nvme0n1p5
           ID-2: swap-1 size: 16.00GB used: 0.00GB (0%) fs: swap dev: /dev/nvme0n1p6
RAID:      No RAID devices: /proc/mdstat, md_mod kernel module present
Sensors:   System Temperatures: cpu: 42.0C mobo: 39.0C gpu: 43.0
           Fan Speeds (in rpm): cpu: N/A
Info:      Processes: 196 Uptime: 56 min Memory: 1661.7/15890.4MB Init: systemd runlevel: 5 Gcc sys: 7.3.0
           Client: Shell (bash 4.4.191) inxi: 2.3.56 
$ cat /usr/share/applications/jetbrains-idea.desktop
[Desktop Entry]
Version=1.0
Type=Application
Name=IntelliJ IDEA
Icon=/opt/idea-IU/bin/idea.png
Exec="/opt/idea-IU/bin/idea.sh" %f
Comment=The Drive to Develop
Categories=Development;IDE;
Terminal=false
StartupWMClass=jetbrains-idea

$ cat /usr/share/applications/jetbrains-pycharm.desktop
[Desktop Entry]
Version=1.0
Type=Application
Name=PyCharm
Icon=/opt/pycharm/bin/pycharm.png
Exec="/opt/pycharm/bin/pycharm.sh" %f
Comment=The Drive to Develop
Categories=Development;IDE;
Terminal=false

Setting python 2.6 as default on Linux

sudo apt-get install python2.6 idle-python2.6
sudo update-alternatives --install /usr/bin/python python /usr/bin/python3.1 1
sudo update-alternatives --install /usr/bin/python python /usr/bin/python2.6 10
sudo update-alternatives --config python

Graphic Card

$ lspci | grep -i amd
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Opal XT [Radeon R7 M265/M365X/M465] (rev 81)

$ dmesg | grep -i radeon
[    5.559181] [drm] radeon kernel modesetting enabled.
[    5.559629] radeon 0000:03:00.0: enabling device (0006 -> 0007)
[    5.570964] radeon 0000:03:00.0: VRAM: 2048M 0x0000000000000000 - 0x000000007FFFFFFF (2048M used)
[    5.570965] radeon 0000:03:00.0: GTT: 2048M 0x0000000080000000 - 0x00000000FFFFFFFF
[    5.571137] [drm] radeon: 2048M of VRAM memory ready
[    5.571138] [drm] radeon: 2048M of GTT memory ready.
[    5.578766] [drm] radeon: dpm initialized
[    5.590312] radeon 0000:03:00.0: WB enabled
[    5.590314] radeon 0000:03:00.0: fence driver on ring 0 use gpu addr 0x0000000080000c00 and cpu addr 0x        (ptrval)
[    5.590315] radeon 0000:03:00.0: fence driver on ring 1 use gpu addr 0x0000000080000c04 and cpu addr 0x        (ptrval)
[    5.590316] radeon 0000:03:00.0: fence driver on ring 2 use gpu addr 0x0000000080000c08 and cpu addr 0x        (ptrval)
[    5.590318] radeon 0000:03:00.0: fence driver on ring 3 use gpu addr 0x0000000080000c0c and cpu addr 0x        (ptrval)
[    5.590319] radeon 0000:03:00.0: fence driver on ring 4 use gpu addr 0x0000000080000c10 and cpu addr 0x        (ptrval)
[    5.591135] radeon 0000:03:00.0: fence driver on ring 5 use gpu addr 0x0000000000075a18 and cpu addr 0x        (ptrval)
[    5.691895] radeon 0000:03:00.0: failed VCE resume (-110).
[    5.691930] radeon 0000:03:00.0: radeon: MSI limited to 32-bit
[    5.691964] radeon 0000:03:00.0: radeon: using MSI.
[    5.691998] [drm] radeon: irq initialized.
[    5.778217] radeon 0000:03:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=none
[    6.717290] [drm] Radeon Display Connectors
[    6.730467] [drm] Initialized radeon 2.50.0 20080528 for 0000:03:00.0 on minor 0

$ lsmod | egrep -i "amdgpu|i915"
amdgpu               2703360  0
chash                  16384  1 amdgpu
i915                 1617920  18
ttm                   106496  2 amdgpu,radeon
i2c_algo_bit           16384  3 amdgpu,radeon,i915
drm_kms_helper        167936  3 amdgpu,radeon,i915
drm                   401408  10 drm_kms_helper,amdgpu,radeon,i915,ttm
video                  45056  1 i915