Count and sort by file extension in folder

One liner to count all files by extension in folders and subfolders.

find . -type f | sed 's/.*\.//' | sort | uniq -c | sort -nr

Grep in utf16 file

A quick way to grep in utf16 by converting it to utf-8 on the fly

iconv -f utf-16 -t utf-8 file.txt | grep query

recursive grep in Powershell

The eqvivalent of grep -r in Powershell

Get-ChildItem -Recurse -Include *.txt -Path . | Select-String -Pattern "foobar"

grep on a remote machine and save output to local machine

Grep on a remote machine and save output to local machine This particular example grep is used for trying to find deleted files on an disk in linux

ssh user@remotehost "grep -a  -P 'searchterm' /dev/md2" > output.txt

Slicing in python

s = "Be yourself; everyone else is already taken."

#First Character
print "{}:  [{}]\n".format("First Character",s[:1])
# First Character:  [B]

#Last character
print "{}: [{}]\n".format("Last character",s[-1:])
# Last character: [.]


#All but first
print "{}: [{}]\n".format("All but first",s[1:])
# All but first: [e yourself; everyone else is already taken.]

#All but last
print "{}: [{}]\n".format("All but last",s[:-1])
# All but last: [Be yourself; everyone else is already taken]

#Practical use removing file extension
f = "Oscar.the writer Wilde.txt"

#Cut filextension with slice
print "{}: [{}]\n".format("Cut file extension with string slice",f[:-4])
# Cut file extension with string slice: [Oscar.the writer Wilde]

#cut file extension with split.
splitlist = f.split(".")
print "{}: [{}]\n".format("Cut file extension with split",".".join(splitlist[:-1]))
# Cut file extension with split: [Oscar.the writer Wilde]

#Cut file extension with regex repl
import re
print "{}: [{}]\n".format("Cut file extension with regex",re.sub("\.\w+$","",f))
# Cut file extension with regex: [Oscar.the writer Wilde]
 
# Std lib way. RTFM :)
import os
print "{}: [{}]\n".format("Cut file extension std lib way",os.path.splitext(f)[0])
# Cut file extension std lib way: [Oscar.the writer Wilde]

Download all APTnotes files from box.com

APTnotes

All improvements are welcome. A pure python or pure bash version would be nice. The reason for the mix is that bash sucks at reading csv files with quotes and lynx is good for resolving javascript links.

'''
A simple crude (15 minutes) script to dowload all aptnotes files from BOX.COM 
The script reads the .csv and uses lynx and grep to find the correct filepath

This script is tested on Ubuntu with lynx

sudo apt-get install lynx

'''

import csv
import os
import subprocess
import shlex
import hashlib

BASEDIR = "dl"

def getsha1(filepath):
    with open(filepath, 'rb') as f:
        return hashlib.sha1(f.read()).hexdigest()

def makedir(dirname):
    if not os.path.exists(BASEDIR+os.sep+dirname):        
        os.makedirs(BASEDIR+os.sep+dirname)

def getlink(linkurl,filename):
    
    if not os.path.exists(filename):
        print "Downloading {}".format(filename)
        dlcommand='lynx -dump  {0}  | grep -Eh --only-matching "https://[^ ]+"\
                                    | grep "/download?shared_link=https://"\
                                    | xargs wget -O "{1}" '.format(linkurl,filename)
        print os.system(dlcommand)
    else:      
        pass
        #print("File already downloaded")

def getcsvfile():
    os.system("wget https://raw.githubusercontent.com/aptnotes/data/master/APTnotes.csv")
 
getcsvfile()

with open("APTnotes.csv", "r") as aptnotes:
    csvreader = csv.DictReader(aptnotes, delimiter=',', quotechar='"')

    for row in csvreader:
        makedir(row.get('Year'))
        filename=BASEDIR+os.sep+row.get('Year')+os.sep+row.get('Filename')+".pdf"
        getlink(row.get('Link'),filename)
        if row.get('SHA-1') == getsha1(filename):
            pass
            #print("File {} is verified by hash".format(filename))
        else:
            print ("Filehash differs {}".format(filename))

print "Done"

Map two lists to a dict in python

Sort tuples in list based on date

keys = ['a', 'b', 'c']
values = [1, 2, 3]
dictionary = dict(zip(keys, values))
print dictionary

#{'a': 1, 'b': 2, 'c': 3}

Sort tuples in a list

Sort tuples in list based on date

import datetime
import pprint

data = [(3, "2012-01-02 12:11:00"),(2, "2012-01-03 12:11:00")\
        , (1, "2014-01-02 12:11:00"), (4, "2011-01-02 12:11:00")]

def get_date(record):
    return datetime.datetime.strptime(record[1], "%Y-%m-%d %H:%M:%S")


def sortitemsondate(items):
    items = sorted(items, key=get_date, reverse=True)
    return items

pprint.pprint(sortitemsondate(data))

# [(1, '2014-01-02 12:11:00'),
#  (2, '2012-01-03 12:11:00'),
#  (3, '2012-01-02 12:11:00'),
#  (4, '2011-01-02 12:11:00')]

Read a file in bash

Read a file line by line in bash

while read line           
do           
    echo $line

done < test.txt   

oneliner version

while read line ; do echo $line ; done < test.txt

Check if process is running in bash script

Check if process is running in bash script

#!/bin/bash

if [ -z "$(pgrep mysql)" ]
 then
     echo "Mysql is not running"     
 else
     echo "Mysql is running"
fi

Backup and restore mysql database

Make backup

mysqldump -u user -p password  | gzip > outputfile.sql.gz

Restore

gunzip < outputfile.sql.gz | mysql -u user -p password databasename

adapted from http://www.ducea.com/2006/10/28/compressing-mysqldump-output/

Compare directory trees

bash version

diff <(cd dir1 && find | sort) <(cd dir2 && find | sort)

Python version soon . . .

Disable line wrap in bash

Disables

printf '\033[?7l'

Enables

printf '\033[?7h'

Brief pcap view with tshark

tshark -T fields -eip.src -e tcp.srcport -eip.dst -e tcp.dstport -e text -r http.cap | sed -e 's/Source GeoIP: Unknown,Destination GeoIP: //g'

Traverse a directory in python

#!/usr/bin/env python
# -*- coding: utf-8 -*- 
import os

for root, dirs, files in os.walk(rootpath):
	for filename in files:
		path = os.path.join(root, filename)
		print path

With file filtering

#!/usr/bin/env python
# -*- coding: utf-8 -*- 
import os
import fnmatch

pattern = "*.conf"
rootpath = "/etc"
for root, dirs, files in os.walk(rootpath):
	for filename in fnmatch.filter(files, pattern):
		path = os.path.join(root, filename)
		print path

Python 2.7 documentation on os

Concatenate files in python

cat folder/* > outfile

import fileinput
with open(outfilename, 'w') as fout:
    for line in fileinput.input(filenames):
        fout.write(line)

Show specific line in large file

Show only line 20 

sed -n '20p' file_name

Show 10 through 20 

sed -n '10,20p' file_name

Show 10 and 20 

sed -n '10p;20p' file_name

File modification time python

import os, time
(mode, ino, dev, nlink, uid, gid, size, atime, mtime, ctime) = os.stat(file)
print "last modified: %s" % time.ctime(mtime)

http://stackoverflow.com/questions/237079/how-to-get-file-creation-modification-date-times-in-python

Update all python pip packages

import pip
from subprocess import call

for dist in pip.get_installed_distributions():
    call("pip install --upgrade " + dist.project_name, shell=True)

i use it with ipython

sudo ipython
""" Cut the code above and type
%paste

Kill process with python on linux

import os
import signal

def kill_process(processname):
    for line in os.popen("ps xa"):
        fields = line.split()
        pid = fields[0]
        process = fields[4]
        print process
        if process == processname:
            os.kill(int(pid), signal.SIGKILL)
            break
        else:
            pass

look here for a windows solution or google it