- Appending data to an existing file
- Reading a file adn altering each one of its lines
- Regular expressions and pattern matching in Go
- Sending information to Unix log files
- Working with dates and times in Go
- Working with user IDs and group IDs
- Learning more information about files and directiories
- Processing log files and extracting useful information from them
- Generating difficult to guess passwords using random numbers
Putting data at the end of a file
We will talk about opening a file for writing without destroying its existing data.
maxwell@MaxwellDBA goproject]$ cat test
[test]: test
: test
[maxwell@MaxwellDBA goproject]$ go run appendData.go test test
[maxwell@MaxwellDBA goproject]$ cat test
[test]: test
: test
test
[maxwell@MaxwellDBA goproject]$ cat appendData.go
package main
import (
"fmt"
"os"
"path/filepath"
)
func main() {
arguments := os.Args
if len(arguments) != 3 {
fmt.Printf("usage: %s message filename\n", filepath.Base(arguments[0]))
os.Exit(1)
}
message := arguments[1]
filename := arguments[2]
f, err := os.OpenFile(filename,os.O_RDWR|os.O_APPEND|os.O_CREATE,0660)
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
defer f.Close()
fmt.Fprintf(f, "%s\n", message)
}
[maxwell@MaxwellDBA goproject]$ Altering existing data
how to modify the contents of a file.
[maxwell@MaxwellDBA goproject]$ vim insertLineNumber.go
[maxwell@MaxwellDBA goproject]$ cat test
a
b
[maxwell@MaxwellDBA goproject]$ go run insertLineNumber.go -init=10 test
Processing: test
Processed 6 lines!
[maxwell@MaxwellDBA goproject]$ cat test
10: a
11:
12:
13:
14: b
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ cat test
a
b
[maxwell@MaxwellDBA goproject]$ go run insertLineNumber.go -init=10 test test test
Processing: test
Processing: test
Processing: test
Processed 18 lines!
[maxwell@MaxwellDBA goproject]$ cat test
22: 16: 10: a
23: 17: 11:
24: 18: 12:
25: 19: 13:
26: 20: 14: b
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ cat insertLineNumber.go
package main
import (
"flag"
"fmt"
"io/ioutil"
"os"
"strings"
)
func main(){
minusINIT := flag.Int("init", 1, "Initial Value")
flag.Parse()
flags := flag.Args()
if len(flags) == 0 {
fmt.Printf("usage: insertLineNumber <files>\n")
os.Exit(1)
}
lineNumber := *minusINIT
for _, filename := range flags {
fmt.Println("Processing:", filename)
input, err := ioutil.ReadFile(filename)
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
lines := strings.Split(string(input),"\n")
for i, line := range lines {
lines[i] = fmt.Sprintf("%d: %s ", lineNumber, line)
lineNumber = lineNumber + 1
}
lines[len(lines)-1] = ""
output := strings.Join(lines,"\n")
err = ioutil.WriteFile(filename, []byte(output),0644)
if err != nil {
fmt.Println(err)
os.Exit(-1)
}
}
fmt.Println("Processed", lineNumber-*minusINIT, "lines!")
}
[maxwell@MaxwellDBA goproject]$ About log files
Log files are important and you should not underestimate the value of the information stored in them. Log files should be the first place to look for help when strange things start happening on a Unix machine.
- first, the output does not get lost, as it is stored on a file
- second , you can search and process log files using Unix tools, such as grep(1), awk(1), and sed(1), which cannot be done when messages are printed on a Terminal window.
About logging
the name of the process is rsyslogd(8), which is an improved and more reliable version of syslogd(8), which was the original Unix system utility for message logging.
The best way to watch one or more log files is with the help of the tail(1)utility, followed by the -f flag and the name of the log file you want to watch. The -f flag tells tail(1) to wait for additional data. You will need to terminate such a tail(1) command by pressing Ctrl + C.
Logging facilities
A logging facility is like a category used for logging information. The value of the logging facility part can be any one of auth, authpriv, cron,daemon, kern,lpr, mail, mark, news, syslog, user, UUCP,local0, local1, local2, local3,local4, local5, local6, and local7.this is defined inside /etc/syslog.conf , /etc/rsyslog.conf.
Logging levels
A logging level or priority is a value that specifies the severity of the log entry. There exist various logging levels including debug, info, notice, warning, err, crit, alert, and emerg, in reverse order of severity.
[maxwell@MaxwellDBA goproject]$ cat useSyslog.go
package main
import (
"fmt"
"log"
"log/syslog"
"os"
"path/filepath"
)
func main(){
programName := filepath.Base(os.Args[0])
sysLog, e := syslog.New(syslog.LOG_INFO|syslog.LOG_LOCAL7, programName)
if e != nil {
log.Fatal(e)
}
sysLog.Crit("Crit: Logging in Go!")
sysLog, e = syslog.New(syslog.LOG_ALERT|syslog.LOG_LOCAL7, "some program!")
if e != nil {
log.Fatal(sysLog)
}
sysLog.Emerg("Emerg: Logging in Go!")
fmt.Fprintf(sysLog, "log.Print: Logging in Go!")
}
[maxwell@MaxwellDBA goproject]$ go run useSyslog.go
Broadcast message from systemd-journald@MaxwellDBA (Mon 2022-12-19 17:18:19 CST):
[2374196]: some program![2374196]: Emerg: Logging in Go!
[maxwell@MaxwellDBA goproject]$
Message from syslogd@MaxwellDBA at Dec 19 17:18:19 ...
journal[2374196]:some program![2374196]: Emerg: Logging in Go!
[maxwell@MaxwellDBA goproject]$ grep "Logging in Go" /var/log/* 2>/dev/null
[maxwell@MaxwellDBA goproject]$ tail -5 /var/log/syslog
tail: cannot open '/var/log/syslog' for reading: No such file or directory
[maxwell@MaxwellDBA goproject]$ Processing log files
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2
/tmp/log.1
error opening file open /tmp/log.1: no such file or directory
/tmp/log.2
error opening file open /tmp/log.2: no such file or directory
[maxwell@MaxwellDBA goproject]$ mkdir /tmp/log.1
[maxwell@MaxwellDBA goproject]$ rmdir /tmp/log.1
[maxwell@MaxwellDBA goproject]$ touch /tmp/log.1
[maxwell@MaxwellDBA goproject]$ touch /tmp/log.2
[maxwell@MaxwellDBA goproject]$ vim /tmp/log.1
[maxwell@MaxwellDBA goproject]$ vim /tmp/log.2
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2
/tmp/log.1
/tmp/log.2
64.183.178.218 31
164.132.161.85 1
66.102.8.135 1
5.248.196.10 1
180.76.15.10 1
66.249.69.40 1
51.255.65.35 31
95.158.53.56 31
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2 | wc
10 18 155
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2 | sort -rn -k2
95.158.53.56 31
64.183.178.218 31
51.255.65.35 31
66.249.69.40 1
66.102.8.135 1
5.248.196.10 1
180.76.15.10 1
164.132.161.85 1
/tmp/log.2
/tmp/log.1
[maxwell@MaxwellDBA goproject]$ go run countIP.go /tmp/log.1 /tmp/log.2 | sort -rn -k2 | head
95.158.53.56 31
64.183.178.218 31
51.255.65.35 31
66.249.69.40 1
66.102.8.135 1
5.248.196.10 1
180.76.15.10 1
164.132.161.85 1
/tmp/log.2
/tmp/log.1
[maxwell@MaxwellDBA goproject]$ cat countIP.go
package main
import (
"bufio"
"flag"
"fmt"
"io"
"net"
"os"
"path/filepath"
"strings"
)
func main(){
minusCOL := flag.Int("COL", 1, "Column")
flag.Parse()
flags := flag.Args()
if len(flags) == 0 {
fmt.Printf("usage: %s <file1> [<file2> [...<fileN]]\n",filepath.Base(os.Args[0]))
os.Exit(1)
}
column := *minusCOL
if column < 0 {
fmt.Println("Invalid Column number!")
os.Exit(1)
}
myIPs := make(map[string]int)
for _, filename := range flags {
fmt.Println("\t\t", filename)
f, err := os.Open(filename)
if err != nil {
fmt.Printf("error opening file %s\n", err)
continue
}
defer f.Close()
r := bufio.NewReader(f)
for {
line, err := r.ReadString('\n')
if err == io.EOF {
break
} else if err != nil {
fmt.Printf("error reading file %s", err)
continue
}
data := strings.Fields(line)
ip := data[column-1]
trial := net.ParseIP(ip)
if trial.To4() == nil {
continue
}
_, ok := myIPs[ip]
if ok {
myIPs[ip] = myIPs[ip] + 1
} else {
myIPs[ip] = 1
}
}
}
for key, _ := range myIPs {
fmt.Printf("%s %d\n", key, myIPs[key])
}
}
[maxwell@MaxwellDBA goproject]$