Linux | 小手册

Ubuntu 虚拟机安装

  • 提前准备:
    • VMware Workshop 15
  • 下载镜像源:中科大发行版镜像源,选取Ubuntu 18.10
  • 兴建虚拟机
    • 加载镜像.ios
    • 配置默认,内存40G
    • 打开,安装默认即可

基础命令

设置环境变量

  • vi ~/.bashrc
    source ~/.bashrc
  • 查看环境配置
    1
    2
    3
    4
    5
    6
    $ uname -a 
    Linux ubuntu18 5.3.0-40-generic #32~18.04.1-Ubuntu SMP Mon Feb 3 14:05:59 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
    $ ldd --version ldd
    (Ubuntu GLIBC 2.27-3ubuntu1) 2.27
    $ gcc --version
    gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0 $ ld --version GNU ld (GNU Binutils for Ubuntu) 2.30

基础安装

  • pip:

共享文件夹

  • 问题看不见/mnt/hgfs下的共享文件夹
    • ll /mnt下面是有hgfs文件夹的,但是ll /mnt/hgfs下面是没共享文件夹的;
    • 输入vmware-hgfsclient可以看到共享文件夹
    • 输入sudo mount -t vmhgfs .host:/ /mnt/hgfs 报送Error: cannot mount filesystem: No such device错误
  • 解决方法:

    sudo vmhgfs-fuse .host:/ /mnt/hgfs -o allow_other,nonempty

开发工具

java

java JDK 是基本的env工具,具体安装过程如下:

  • installation : sudo apt-get install default-jre default-jdk
  • configuration env:
    • vi ~/.bashrc # vim bashrc file
    • export JAVA_HOME=/usr/lib/jvm/default-java # add to environment # add to bashrc
    • ‘source ~/.bashrc # the configuration take effect’
  • test:
    • echo $JAVA_HOME
    • java -version

测试结果为:
java

需要注意的是:

  1. vim 版本问题解决:
    sudo apt-get update
    sudo apt-get install vim-gtk
  2. export 等号前后不能有空格

Hadoop

安装分布式Hadoop 3.12.0 在服务器上

Scala

  • 下载scala安装包,包的最新版本可参考scala官网,截止2019.11.20 最新为2.13.1

    1
    2
    3
    4
    wget https://downloads.lightbend.com/scala/2.13.0/scala-2.13.0.tgz # 下载包
    sudo tar -zxvf scala-2.13.0.tgz -C /usr/local # 解压缩
    sudo mv ./scala-2.13.0/ ./scala # 修改名字
    sudo chown -R root ./scala # 配置权限
  • 配置环境变量

    1
    2
    3
    4
    vi ~/.bashrc # 向配置文件中加入路径
    export SCALA_HOME=/usr/local/scala
    export PATH=$PATH:$SCALA_HOME/bin
    source ~/.bashrc # 配置生效
  • 检验

    • scala -version 查看版本安装
      scala -version
    • scala 运行scala
      scala

Spark

安装分布式spark

Sublime

  • install gpg key

    1
    wget -qO - https://download.sublimetext.com/sublimehq-pub.gpg | sudo apt-key add -
  • select channel

    1
    sudo apt-get install apt-transport-https
  • update and install

    1
    2
    3
    echo "deb https://download.sublimetext.com/ apt/stable/" | sudo tee /etc/apt/sources.list.d/sublime-text.list
    sudo apt-get update
    sudo apt-get install sublime-text
  • usage:subl a.txt

服务器

ssh建立服务器联系

以网易云为例,建立多个节点的服务器联系。

  • 创建多台服务器和申请ip,绑定主服务器和ip address
    ipbing
    ipbing
    ipbing
  • ssh连接
    • ssh-keygen -t rsa # genertate key pairs # generate
    • cd ~/.ssh
    • 添加ssh密钥在云服务器上,复制pubkey(本地到hadoop1 主节点,其他增加hadoop1
      1 的pub key)
      ssh
      ssh
    • 本地登陆 ssh root@ip_hadoop1, 从hadoop 1 SSH登陆hadoop其他节点
  • 通过配置hosts文件建立直接hostname的连接方式
    sudo vi /ect/hosts # configure hosts
    10.173.32.21 hadoop1 # 添加hostname和节点
    scp -r /etc/hosts root@hadoop2:etc # send to other nodes
    传送host到别的节点
  • 刷新hostname
    ssh-keygen -f "/root/.ssh/known_hosts" -R hadoop2
    ssh成功连接

换镜像

  • 用vmware上外网一直失败,原因不明,所以有一个国内的镜像十分重要:

做法如下:

  • 先找到apt下载源的配置文件/etc/apt/sources.list
  • 备份cp /etc/apt/sources.list /etc/apt/sources_bak.list
  • 修改镜像源:
    • 注意换镜像源的时候用lsb_release -c得到系统版本代号(comsic,bionic,precise,trusty,vivid,wily,xenial等)
    • 以下提供了阿里云的镜像,这里发现comsic可以使用bionic没问题,ali没有comsic的版本,清华也没有
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
# deb cdrom:[Ubuntu 18.04.4 LTS _Bionic Beaver_ - Release amd64 (20200203.1)]/ bionic main restricted

# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution.
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic-updates main restricted

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team. Also, please note that software in universe WILL NOT receive any
## review or updates from the Ubuntu security team.
deb http://mirrors.aliyun.com/ubuntu/ bionic universe
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic universe
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates universe
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic-updates universe

## N.B. software from this repository is ENTIRELY UNSUPPORTED by the Ubuntu
## team, and may not be under a free licence. Please satisfy yourself as to
## your rights to use the software. Also, please note that software in
## multiverse WILL NOT receive any review or updates from the Ubuntu
## security team.
deb http://mirrors.aliyun.com/ubuntu/ bionic multiverse
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates multiverse
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic-updates multiverse

## N.B. software from this repository may not have been tested as
## extensively as that contained in the main release, although it includes
## newer versions of some applications which may provide useful features.
## Also, please note that software in backports WILL NOT receive any review
## or updates from the Ubuntu security team.
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
# deb-src http://cn.archive.ubuntu.com/ubuntu/ bionic-backports main restricted universe multiverse

## Uncomment the following two lines to add software from Canonical's
## 'partner' repository.
## This software is not part of Ubuntu, but is off# deb http://archive.canonical.com/ubuntu bionic partner
# deb-src http://archive.canonical.com/ubuntu bionic partner

deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted
# deb-src http://security.ubuntu.com/ubuntu bionic-security main restricted
deb http://mirrors.aliyun.com/ubuntu/ bionic-security universe
# deb-src http://security.ubuntu.com/ubuntu bionic-security universe
deb http://mirrors.aliyun.com/ubuntu/ bionic-security multiverse
# deb-src http://security.ubuntu.com/ubuntu bionic-security multiverse
  • 更新apt源sudo apt-get upgrade

ppa:

1
2
sudo apt-add-repository -r ppa:armagetronad-dev/ppa
sudo apt update -q

reference

[^1]:fuhanlin’s blog

Comments

Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now

×